Benchmarking IBM Cloud Pak for Integration installation experience

Company
IBM

Year
2021

Research objectives:

  • Identify, quantify, prioritize key blockers to installing CP4I

  • Provide recommendations for improving task success, satisfaction, and reducing time on task.

  • Create a process for communicating and managing fixes among cross functional teams (design, content writing, marketing and engineering teams respectively)

Context:

At IBM, our cloud strategy meant a more consolidated product experience was required. There were gaps in the CP4I (Cloud Pak for Integration) installation experience that caused significant customer dissatisfaction. While stakeholders were aware of the frustration caused by the installation process, there was no actionable plan to address this.

After successfully completing several other benchmarking projects at IBM, I was requested to take on this project by the executive team to tackle fundamental UX issues at the cross-product level in alignment with the desire to make IBM more accessible out of the box. 

UXR methods I applied:

  • Heuristic evaluation — an expert walkthrough of the product experience without participants. This method helped me to understand the user flow through both command line and the graphical interface. It helped to predict key issues before testing with participants.

  • CIF Benchmarking (Common Industry Format) 90 minute sessions — the timing of this was determined by an expert completion on each task multiplied by 3 or 5 to give participants sufficient time to attempt each task.

  • Prioritization workshop — I applied this method after synthesizing the research data. This prioritized issues with higher severity as well as taking into account the team’s current and future pipeline of work.

"I quickly realized that through using the online documentation alone, it was impossible to successfully complete the process."

I created a spreadsheet that helped to track time on task and log data as the sessions were run. Through tracking task start and end time, I was able to calculate task duration. I also noted happy paths and where participants deviated to calculate a metric called ‘lostness.’ Participants were allowed to ask for help if they were struggling in the task and this was noted as an assist.

Later, as I reviewed the task footage, I also noted the number of errors. Finally I looked at task success rate, a crucial metric that showed whether participants could successfully complete their set out task. As I ran sessions, I watched participants consistently to struggle to complete certain tasks. As frustrating as it was to watch, this was crucial footage that motivated our internal teams to make changes.

I started with a heuristic evaluation and walked through the installation experience in both graphical user interface (GUI) and command line (CLI). Pulling screenshots into a mural whiteboard, and using stickies to highlight blockers. I started parsing out tangible steps in the installation process that I would mark as

I quickly realized that it was impossible to successfully complete the process through using the online documentation alone. Reaching out to one of our CP4I installation experts, who was responsible for onboarding customers, I found out that there were several additional crucial pieces of information that were undocumented.

I formatted issues and created a process aimed to serve multiple stakeholders in a single repository. Using Github with a Zenhub integration, I titled issues and tagged them to be easily searched and used by the various cross-functional teams that would be responsible for each fix — Design, Engineering, Marketing, Documentation.

An example of a Github board with Zenhub integration use to store and assign UX issues to the relevant owners. 

I considered the best way to share the rich data that we collected as I wanted a consolidated high level view to show areas that needed the most attention. I created a visual artifact as a table with individual graphs for different metrics.

I displayed the average time on task and where they exceeded the cutoff time, these metrics correlated with lower task completion success and higher number of errors.

We could see at a glance several tasks that were particularly unsuccessful with CLI methods being especially problematic. This was a big issue as our core customers were developers, IT admins who generally prefer using CLI for its ease of use and integration.

Task completion shows the percentage of participants able to successfully complete a task with <2 assists and within the cut-off time. (The cut-off time was determined by 3x expert time on task, and made with an assumption that in real life, participants would likely give up before spending the allocated time)

"We knew there were major problems but no one has ever articulated the issues in a way to get them resolved.”

- Engineering team

I managed each team to work on ownership of specific issues:

  • Setting up regular check ins to track issues and determining how we would work together moving forward and how to access and use the board

  • Acted as a soundboard for potential solutions as well as reporting back progress to leadership. 

Impact

  • 60+ issues identified, tagged and assigned with 80% resolved within 3 months resulting in hugely improved customer usability (to be remeasured in 1 year)

  • High visibility from multiple cross-functional partners including marketing, documentation, customer success, development increasing requests for UXR at IBM, promotion and bonus from our head of Design.

For questions on how I set up this project and going into details, please reach out!