Why conduct a usability test?

Now that you've chosen your flow, you can conduct a usability test to make sure it provides a good experience for your contacts. Usability tests keep your focus on your contacts' experience by assessing how easy your flow is to use and learn. This step lets you make the best possible impression when you send out the final version of your flow.

Usability test requirements

A large group of test contacts representative of your target population.
A clear and precise protocol for your test contacts to follow. This might include a pre-test questionnaire, the test itself, and a post-test questionnaire.
A point of contact who is trained to offer instructions and answer questions. In your pilot test, the facilitator took a more hands-on approach. In your usability test, the facilitator takes a supporting role while your test contacts take the lead.
Observers who track the behavior of the test contacts.

Elements of a usability test plan

Subjective metrics

Background questions. Ask these questions before the test.

Ease of use, comfort, and satisfaction. Ask these questions after each task is completed.

Overall ease of use, comfort, and satisfaction. Ask these questions when the test is completed.

Likes, dislikes, and recommendations. Ask test contacts what they liked most about your service, what they liked least, and whether they have recommendations for improving it.

Continued use. Measure how likely your test contacts are to keep using your service after the test is completed.

Net promoter score. On a scale from extremely unlikely to extremely likely, measure how likely your test contacts are to recommend your service to (1) a family member, (2) a friend, and (3) a colleague.

Quantitative metrics

Flow completion. A flow is completed when a contact has successfully passed through each step, whether it's an action or a split step.

Critical errors. Critical errors leave your contacts unable to finish a flow. For example, a contact might consistently respond with the wrong answer, an incorrectly formatted answer, or an answer you haven't accounted for, or might opt out.

Non-critical errors. Non-critical errors are errors that the contact can recover from. The most common one you'll see is a response that is categorized as "other," causing the contact to be redirected to a split step. These errors make the flow less efficient to complete. Keep a tally of the number of non-critical errors that occur per contact, per contact per flow, per flow, and overall.

Error-free rate. The error-free rate is the percentage of contacts who complete the flow(s) without experiencing or committing any errors.

Time on task. The amount of time it takes the contact to complete the flow(s): the time a flow was completed minus the time it was started.

Test results

Analyze what you've observed. Isolate the flows that had the highest and lowest completion rates. If possible, include a summary of the completion rates by contact, step, and flow. Depending on the metrics you collected, you may want to show:

The number and percentage of contacts who completed each flow or series of flows.
The average time taken to complete each flow or series of flows by those who completed them.
Ease of use, comfort, and satisfaction results.
Your contacts' comments, if they are illustrative.

Findings and recommendations

List your findings and recommendations based on all your data. Each finding should have a basis in what occurred during the test. Sort your findings and recommendations by flow. Finally, identify positives as well as negatives, since what's working well should be kept as you move forward.

Implement and retest

For a usability test to have value, you'll need to use what you've learned to improve your service. You may not be able to implement all recommendations or requests. Developing anything is a series of trade-offs in which you balance factors such as budget, timeframe, and other external constraints. If you can't implement all the recommendations, set priorities based on the most common problems. As you prioritize, push to get the changes your contacts want.