Why conduct a usability test?

Why conduct a usability test?

Now that you've chosen the perfect flow, you can conduct a usability test to make sure it provides a good experience for your contacts. Usability tests encourage you to focus on your contacts’ experience by assessing ease of use and learnability. This step is important - it allows you to make the best possible impression when you send out the final version of your flow.

Usability test requirements

A large group of test contacts representative of your target population.

A clear and precise protocol for your test contacts to follow. This might include a pre-test questionnaire, the test itself, and a post-test questionnaire.

A point of contact who is trained to offer instructions and answer questions. In your pilot test, the facilitator took a more hands-on approach. In your usability test, the facilitator will take a supporting role while your test contacts take the lead.

Observants who will track the behavior of the test contacts.

Elements of a usability test plan

Subjective metrics

Background questions. Ask these questions prior to the test.

Ease of use, comfort and satisfaction. Ask these questions after each task is completed.

Overall ease of use, comfort, and satisfaction. Ask when the test is completed.

Likes, dislikes and recommendations. Ask test contacts what they liked most about your service, what they liked least about your service, and if they have recommendations for improving it.

Continued use. The likelihood your test contacts will use your service when the test is completed.

Net promoter score. On a scale that ranges from extremely unlikely to extremely likely, measure the likelihood that your test contacts would recommend your service to (1) a family member (2) a friend and (3) a colleague.

Quantitative metrics

Flow completion. The flow is completed when a contact has successfully passed through each step - be it an action or split step.

Critical errors. Critical errors result in your contacts being unable to finish a flow. For example, consistently responding with the wrong answer, an incorrectly formatted answer, an answer you haven’t accounted for, or an opt-out.

Non-critical errors. Non-critical errors are errors that the contact can recover from. The most common error you’ll experience is a response that is categorized as “other,” causing the contact to be redirected to a split step. These errors result in the flow being completed less efficiently. You’ll want to keep a tally of the amount of non-critical errors occur per contact, per contact per flow, per flow, and overall.

Error-free rate. The error-free rate is the percentage of contacts who complete the flow(s) without experiencing or committing any errors.

Time on task. The amount of time it takes the contact to complete the flow(s) - the time a flow was completed minus the time it was started.

Test Results

Construct an analysis of what you’ve observed. Isolate the flows that had the highest and lowest completion rates. If possible, include a summary of the completion rates by contact, step, and flow. Depending on the metrics you collected you may want to show the:

Number and percent of contacts who completed each flow or series of flows.

Average time taken to complete each flow or series of flows for those who completed them.

Ease of use, comfort, and satisfaction results.

Include your contacts’ comments if they are illustrative.

Findings and Recommendations

List your findings and recommendations using all your data. Each finding should have a basis in what occurred during the test. You’ll want to sort findings and recommendations by flow. Finally, it’s important to keep in mind that identifying positives is equally as important as identifying negatives - what’s working well should be maintained moving forward.

Implement and Retest

For a usability test to have value, you’ll need to use what you’ve learned to improve your service. You may not be able to implement all recommendations or requests - developing anything is a series of trade-offs in which you balance a number of factors, such as budget, timeframe, and a handful of external factors. If you can’t implement all the recommendations, assign priorities based on the most common and prevalent problems. As you prioritize, push to get the changes your contacts want.

Updated on: 30/11/2021