Use Optimizely to A/B Test Twilio IVR Flows

Use Optimizely to A/B Test Twilio IVR Flows

Optimizing IVR flows can be difficult, but with A/B testing tools such as Optimizely, you can optimize Twilio IVR flows through experimentation.

In this article we will show you how to test a longer, more descriptive voice message versus a shorter, more direct one for customers picking up curbside at a store. To measure the effectiveness of the long versus short message, we will also discuss how to capture survey results and send them to Optimizely.

Follow along to learn how to configure a Twilio Studio flow that splits users based on how Optimizely’s SDK buckets them for an experiment. You will also learn how to send Optimizely tracking metrics such as ‘Completed IVR’ to see what score a user would give your service.

Tutorial Requirements

• A Twilio account. If you don’t yet have a free account, you can sign up here.
• An Optimizely Full Stack project -- if you are new to Optimizely you can create a free trial of Full Stack here.

Optimizely Configuration

To get started, we will set up our Optimizely project and create our experiment.

To start, sign on to Optimizely to create an account. Create Full Stack project as seen below:

Create a new A/B test by clicking on the Create New… button in the upper right hand side.

Give your Experiment Key the name "IVR_SCRIPT". Keep in mind that the string we input for the Experiment Key section in Optimizely is what we will need for our Twilio Studio flow. Feel free to give your experiment whatever Experiment Name or Description that makes sense for you. Typically the “Description” is a good place to put your hypothesis.

Once you click Create A/B Test you will be redirected to that experiment’s dashboard. Rename the “Variation Keys” SHORT and LONG and save your changes.

Click on the Metrics tab, and add a new event. One your first event, give it the name "COMPLETED_IVR" and add it as a metric to the experiment as an “Increase in total conversions per visitor”.

Create two more events called “POSITIVE_REACTION” and “NEGATIVE_REACTION”  and add it as a metric to your experiment as an “Increase in total conversions per visitor”.

Save your changes. These settings are important to measuring our events on your Optimizely results dashboard.

Click the Run button to start your experiment in your development environment. It doesn’t matter which environment you start your experiment in if this is for tutorial purposes, however when you start publishing your experiments in a production environment, Optimizely has environment features to separate the state of experiments based on their readiness.

Twilio Function Configuration

Next, we will set up the Twilio Functions which will allow you to use the Optimizely SDKs to split traffic and send events to Optimizely for reporting purposes.

In the Twilio Functions console, add a new service and give it a name like “ab-testing”. Create 2 new endpoints within that service - one called split-traffic and one called track-event.

Add the following dependencies along with their version numbers. The screenshot shows the module versions at the time this article was written:

• Module: @optimizely/optimizely-sdk
• Module: axios
• Module: crypto
• Module: twilio

Copy your Optimizely SDK key from your Development environment which can be found in your Settings tab. Under your Twilio Function dependencies, add a new dependency called OPTIMIZELY_KEY and paste in the Optimizely SDK key.

In your split-traffic function add the following code [see GitHub link]

The Optimizely SDK is used to initialize an Optimizely client and split traffic in this function. This will ultimately send an ‘impression’ event to Optimizely via the .activate() method. This will track whether the user was in the experiment and what variation they were in.

In your track-event function add the following code [see GitHub link]

You will notice we’re using a crypto library to hash phone numbers. This is because Optimizely considers phone number PII and contractually does not allow customers to store them in their databases. Keep this in mind if you decide to use Optimizely to experiment on other Twilio products.

Save and deploy your functions and copy the URLs for both the split-traffic and track-event functions. You will need these 2 URLs to configure your Studio Flow to your specific Optimizely instance.

Studio Flow Configuration

Finally, we’re going to create a Studio IVR flow. Studio lets us set up communications flows using a simple drag & drop interface.

To get started with this step, visit the Studio console here.

Create a new Studio flow, and choose Import from JSON. Then copy and paste the JSON file in the following GitHub to create your AB testing studio flow [see GitHub link]

Your Studio Flow should look like this:

Click into the split_based_on_optimizely widget, here you will see a couple of parameters:

• From which is the phone number that is called in and used as the Optimizely User ID.
• experiment_name which should match the experiment key in your Optimizely dashboard.

Keep in mind that experiment keys are case sensitive. We’ve constructed the demo in a way that you can use these functions again for other experiments. For example, if you wanted to test SMS messages, you can reuse this same split-traffic function and change the experiment_name parameter to a new experiment key that you have created.

In the settings of the function_split_traffic widget, remove the current URL from the Request URL field, and paste your split-traffic function URL. This will ensure that your Optimizely account will receive events.

We used this same design for tracking Optimizely events within Twilio too. The track-event has a parameter called event_name which should match the event key you created in Optimizely.

This function also requires the From parameter which is the user’s phone number used in the split-traffic function. It is imperative that the From value here matches the From value from the split-traffic widget as these are used to tell Optimizely what User ID should be used for results attribution.

If these inputs stay the same, Optimizely will be able to consistently track the event to what variation the user was exposed to. These functions can also be reused and take in different event key names for your future experiments.

This Studio Flow prompts the caller to rate their experience on the phone and uses that rating to track either a POSITIVE_REACTION or NEGATIVE_REACTION event in Optimizely. As more people call in and go through the experiment, the Optimizely results page will show you which message will give people a positive or negative response.

Click into the function_track_IVR_COMPLETED widget and replace the Request URL with the track-event URL from the Twilio Functions console. Replace the Request URL with your track-event function URL as well for both the function_track_NEGATIVE and the function_track_POSITIVE widgets.

Once your Studio Flow is set up, search for and purchase a Twilio number.  Under the Voice section, set incoming calls to route to your A/B testing Studio Flow.

Test out the project

Now that you’re set up, give your Twilio number a call and see what variation you’re in.

After you’ve completed going through the IVR flow, check your Optimizely results page for your experiment and check that your events are being recorded correctly.

Did everything work? Awesome! If it didn’t, don’t worry - I’ve listed some of the questions you might ask below.

Troubleshooting and FAQs

I’m not seeing any visitors/events in my Optimizely results page

Currently there is an Optimizely SDK logger that is configured in all of three of the functions. Navigate to the split-traffic function and enable live logs. Try running the studio flow again and see if Optimizely gives you any additional info.

The Optimizely results page updates every 5 minutes. Make sure your results page is updated to a time that would be after the event was sent.

Double check that your Optimizely experiment key matches your Twilio function experiment_name parameter. This is case sensitive.

Can I add more variations?

Yes! Feel free to add more variations to this particular experiment or future experiments. You will want to update the “Split” widget to make sure you are accounting for what message or logic should happen for the additional variations.

Can I use a different A/B testing tool?

Yes! This project is meant to be a jumping off point for how to best A/B test in Twilio. Most other server side testing solutions will require similar inputs (User ID and experiment name). Check their documentation to make sure you have everything you need to replace the Optimizely SDKs.

A/B Testing Twilio IVR Flows with Optimizely

Awesome! Now that you’ve got an A/B testing setup, try running an experiment of your own. We can’t wait to see what you test!

Heather is a Senior Solutions Engineer at Twilio who was recently a Solutions Engineer at Optimizely where she cultivated her passion helping people build great products through experimentation. She can be reached at hhargreaves@twilio.com.