A/B Testing your Flows
Within the platform, you can use the Split Randomly action to create equally distributed pathways through which your contacts will randomly pass, enabling you to test the efficacy of your flows. We also provide a handy simulator on the same page you build your flows to allow you to test them as you build.
Outside the platform, we encourage you to send real messages to measure flow performance and conduct A/B tests.
A/B testing (also called split or multivariate testing) is a method you can employ to test the effectiveness of two or more versions of a single message, flow or campaign with respect to a desired outcome. It's a type of controlled experiment conducted on similar audiences, the initial version being the control and any additional versions being treatments.
Define objectives: What are you trying to achieve with your flow, distribute information, collect data or something else?
State your hypotheses: Which version of your flow do you think will perform best? Why?
Create a list of the things want to test: For example, flow length, message length, vocabulary and method of engagement (will the contact prompt the flow through a keyword or phone call, or will the flow start on a schedule?).
Let’s say version A addresses your contacts by name while version B doesn’t. When testing two versions of a flow, make sure you’re only measuring the impact of a few changes at a time. If version A is too dissimilar to version B, you’ll have trouble isolating the variable responsible for the result.
Show version A and version B to similar groups and ensure each is representative of the entire target population. If you show version A to only experienced contacts and version B to only new contacts, your results may be skewed. Ideally, you want to split your target population evenly.
One of the best things about the platform is time to deployment. You can sign-up, connect a channel and send messages in as little as 5 minutes if you want to, but this doesn’t mean everyone should take that approach. Being that SMS is just as much an extension of an organization's brand as email, larger deployments need to conduct tests to make sure flows are well-received and driving results.
Creating a new version of a flow is simple: click the ☰ menu icon in the flow editor and select the “Copy” option. This will produce an exact copy of the flow for you to edit and rename.
You can link your versions using a Split Randomly action.
In addition to the primary marker of success you select when you identify your objectives, we provide a completion rate for each flow represented by a percent value. The completion rate allows you to see how many of the contacts that have entered the flow have finished it. Comparing the completion rates of two versions of a flow is easiest way to measure effectiveness. RUNS/COMPLETION
Outside the platform, we encourage you to send real messages to measure flow performance and conduct A/B tests.
A/B testing (also called split or multivariate testing) is a method you can employ to test the effectiveness of two or more versions of a single message, flow or campaign with respect to a desired outcome. It's a type of controlled experiment conducted on similar audiences, the initial version being the control and any additional versions being treatments.
Getting Started
Define objectives: What are you trying to achieve with your flow, distribute information, collect data or something else?
State your hypotheses: Which version of your flow do you think will perform best? Why?
Create a list of the things want to test: For example, flow length, message length, vocabulary and method of engagement (will the contact prompt the flow through a keyword or phone call, or will the flow start on a schedule?).
Conducting an A/B test
Test one item at a time.
Let’s say version A addresses your contacts by name while version B doesn’t. When testing two versions of a flow, make sure you’re only measuring the impact of a few changes at a time. If version A is too dissimilar to version B, you’ll have trouble isolating the variable responsible for the result.
Send the flow to similar representative samples.
Show version A and version B to similar groups and ensure each is representative of the entire target population. If you show version A to only experienced contacts and version B to only new contacts, your results may be skewed. Ideally, you want to split your target population evenly.
Be patient.
One of the best things about the platform is time to deployment. You can sign-up, connect a channel and send messages in as little as 5 minutes if you want to, but this doesn’t mean everyone should take that approach. Being that SMS is just as much an extension of an organization's brand as email, larger deployments need to conduct tests to make sure flows are well-received and driving results.
Creating flow versions
Creating a new version of a flow is simple: click the ☰ menu icon in the flow editor and select the “Copy” option. This will produce an exact copy of the flow for you to edit and rename.
You can link your versions using a Split Randomly action.
Evaluating the results of an A/B test
In addition to the primary marker of success you select when you identify your objectives, we provide a completion rate for each flow represented by a percent value. The completion rate allows you to see how many of the contacts that have entered the flow have finished it. Comparing the completion rates of two versions of a flow is easiest way to measure effectiveness. RUNS/COMPLETION
Updated on: 31/05/2023
Thank you!