After testing your flows in the simulator, you’re ready for the second step in our recommended testing protocol: the pilot test. A pilot test, or "pilot," is your first trial run: a small-scale version of your larger project, and arguably the most important step in testing your SMS program. Your SMS program will be an automated system made up of multiple components (contacts, phones, carriers, channels, and flows). It also represents your project, so it's best to test each component thoroughly.

A pilot allows you to:

• Make sure messages are being delivered to each major carrier in your country.

• Get a good sense of how long it will take your channel or carrier to deliver and receive messages.

• Give your team practice facilitating tests.

• Evaluate the clarity of your questions and flow logic from your test contacts’ point of view.

• Make last-minute adjustments (carrier, connection method, flow structure/content, etc.).

• Determine whether you’re ready to increase scale.

Pilot requirements

1. A group of 5-10 independent test contacts who represent your target population.

2. The current version(s) of your flow(s).

3. One or more pilot facilitators. Facilitators conduct the pre- and post-pilot evaluations as well as the test itself.

4. Observers. Observers watch the test contacts’ responses through the dashboard, along with their overall behavior. See "Best practices" below.

During a pilot, it’s best to be present with your test contacts. If your flows are scheduled using a campaign, or you want your contacts to respond asynchronously (throughout the day or week, in their own time), you can make up for it by checking in with them at the start or end of each day. Run the pilot 3-5 days before your usability test so that you have time to deal with any technical issues or make changes to your scenarios or materials.

Things to look for:

• Do the test contacts understand the objective of the flow/campaign?

• Do the test contacts feel comfortable responding to your questions and/or performing your tasks?

• Is the wording of your flow(s) clear?

• Are your contacts being sorted into the right groups? Could you do a better job of sorting them?

• Are you properly categorizing responses?

• Are the answer choices compatible with the test contacts’ experiences?

• Do any of the items require them to think too long or hard before responding? If so, which ones?

• Do any steps produce irritation or confusion?

• Which steps are receiving the most “other” responses?

• Do the answers collected satisfy your objectives?

• Are your flows too long?

• According to your test contacts, has anything been overlooked?

Best practices

Remain neutral. If a test contact asks a question, reply “What’s your best guess?” Don’t lead your test contacts. If a test contact gives up, you’ll need to decide whether to provide a hint or end the test.

Focus your observations. Give productive and unproductive paths equal attention. Observers should record what the test contacts do in as much detail as possible, as well as what they say, in their own words. The more you understand about your contacts’ SMS behavior, the more effective your SMS program will be.

Measure both performance and preference. People’s performance and preferences don’t always match, especially when it comes to their mobile phones. Contacts often perform poorly even though their subjective ratings are high. Conversely, they may perform well but give your program a poor subjective rating.

• Quantitative metrics include completion rate, time to completion, errors (“other” responses), opt-outs, etc.

• Subjective metrics include test contacts’ self-reported satisfaction and comfort ratings.

Evaluating a pilot

At the end of a pilot, you should be able to answer the following questions:

1. Was the test group’s overall reaction positive or negative? The test group’s feedback can help confirm whether your program is a good fit for your population and whether minor changes to the program are appropriate or necessary.

2. Are you allocating your time and resources properly? The pilot will help you determine whether you need to spend more time or resources on particular aspects of your program. For example, you might learn that you need to change your method of engagement, flow length, or flow timing.

3. Does your evaluation strategy need improvement? Use the pilot to test your evaluation method as well. Are there metrics you’d like to have that you aren’t collecting? The pilot gives your evaluation and implementation teams a chance to work together before you increase scale, and to troubleshoot any logistical issues with distributing and collecting evaluation data.

4. Are you ready to increase scale? A pilot can reveal challenges that might arise during a larger-scale implementation and make sure your team is prepared to handle the issues that can come with an increase in scale. The answer largely depends on your answers to the other questions.