The Definitive Guide to SMS Conversation Tracking

The Definitive Guide to SMS Conversation Tracking

SMS is a fantastic way to let your customers easily communicate with your company or each other. It’s a familiar medium that’s a native part of virtually every mobile phone and allows both one-way and bi-directional communication.

However, whether you are building the next WhatsApp or an SMS-based workforce automation system, apps that require multiple messages to flow between your application and a user means you’re likely going to need to figure out how to track application state between those messages. To help you do this Twilio offers a simple solution that lets you store state associated with a message conversation: cookies.

In this post I’ll show you how you can leverage HTTP cookies to track an SMS conversation. I’ll start by explaining how cookies work both in a common web browser/web server scenario and then show you what happens when Twilio enters the picture. Finally I’ll show you how to build a simple conversation tracking app.

Come to the Dark Side, We Have Cookies

A cookie is a small piece of data that is sent from a website (the ‘server’) and stored by the browser (the ‘client’). Let’s start by looking at how cookies work in a typical web browser / web server scenario. Assume for this example that this is the first time that the web browser has ever made a request to a particular web server. Here is what the HTTP request/response process looks like:

The client (a web browser) has made a single GET request to the server for ‘index.html’. When the server responds it can return as part of its response one or more cookies, each of which is defined using the Set-Cookie header as a single key/value pair. In this example the server has included two different cookies in its response, one which uses the Expires parameter to explicitly set how long this cookie should be valid.

For each cookie in the response the client creates an association between the cookie and the domain of the website that provided it. Creating this association is important because it lets the browser know which cookies it needs to include the next time it makes a request to the same domain.

Now that the server has sent cookies to the client, the client is free to modify the value of the cookies and on subsequent requests to that server, can send those cookies. In the example below the client is sending the two cookies it received back to the server by including the Cookie header in its HTTP request:

Note how the client has taken the values from each Set-Cookie header, combined them into a single semi-colon delimited string and set that string as the value of the Cookie header. This is because the HTTP specific states that only one Cookie header is allowed in an HTTP request, whereas an HTTP response can include many Set-Cookie headers..

The ability for the client to persist cookies and exchange them with the server on each request and response allows for a client that can keep state and communicate that state to the server.

Now let’s look more closely at how this same process works when we swap out the web browser from our example and in its place make Twilio the HTTP client.

How Twilio Uses Cookies

When a user sends a text message to a Twilio phone number, Twilio makes an HTTP request to the URL that you’ve configured for that phone number’s Sms Request URL. That URL typically is an web application that you’ve created whose job it is to do something with that inbound SMS message and then optionally return some TwiML commands back to Twilio.

Because Twilio makes HTTP requests just like a web browser does, the interaction between Twilio and your web server looks really similar to what I’ve already shown.

There are two differences between how a browser stores cookies and how Twilio does. First, unlike a browser which supports both session and persistent cookies, Twilio treats all cookies as persistent, regardless of the absence or presence of the Expires attribute. All cookies sent to Twilio will expire after four hours, though you can use the Expires header to set this to a shorter time. Second, unlike a browser which associates a stored cookie with a domain name, Twilio associates each cookie with the To/From phone number pair of the inbound SMS message.

As Twilio receives additional inbound SMS messages to a specific To/From phone number pair, we will check to see if we have cookies stored for that pair. If cookies exist we will send them in the HTTP request to your web app.

Building a Simple Conversation Tracking Application

Now that I’ve shown you how cookies work in a typical HTTP request/response, lets get to building an Twilio app that leverages cookies. I’ll start by showing you how to build a really simple application uses cookies to keep track of how many messages have occurred in a single SMS conversation.

To build the application, I’m going to use Flask, an easy to use Python microframework. I’ll also use the Twilio Python helper library which makes it really easy to generate the TwiML I’ll return to Twilio when a user sends me a text message. Finally, if you are following along at home and do not yet have a Twilio account, now would be a great time to go sign up for a free trial account. Of course if you want jump ahead and just grab the code for this first sample, it’s all available in this Gist.

The first step to create my application is to import the different libraries I’m using in this app and configure the Flask framework:

Next I need to create a route that Twilio can make HTTP requests to in order to let me know that it has received a text message to my Twilio phone number:

Once I have the route created I can log into my Twilio account and set this route as the Message Request URL for my Twilio phone number:

I’ll start by using Flask to grab the value of a cookie named messagecount from the incoming HTTP request and increment it by one:

Notice that if the cookie doesn’t exist, which we know it won’t on Twilios first request to my app, I just default the counter to zero.

Next using the Twilio Python helper library I create the TwiML response I want to send back to Twilio. In this case I’m sending back the verb to tell Twilio to send an SMS message right back to to the person who just sent one to me.

Finally I create a new HTTP response using the TwiML as the response content and then add a new cookie to it using the set_cookie function:

In this case I am creating a persistent cookie by calculating a new expiration time that is four hours in the future and then including that in the cookie by specifying the expires argument.

And thats it. Twilio will store this cookie for me and associate it with the unique To/From phone number pair that sent the message. The next time that phone number sends me a message (assuming its within 4 hours), Twilio will include the cookie in its request to my app, allowing me to increment the value by one.

And because Twilio associates the cookie with a specific To/From phone number pair, if multiple users are sending messages to my Twilio phone number, each of those conversations will maintain its own unique messagecount cookie.

Tracking Large Object Graphs using Cookies

There are times where you need to store larger amounts of transient data then cookies are appropriate for. In those cases one option is to use session, which is a feature most web frameworks including Flask offer.

Session works by allowing you to store transient data either in a digitally signed cookie, or more commonly in a server side data store like a database. When stored on the server the web framework uses a signed cookie to store a unique session identifier, which it can use to locate the session data in the store on each subsequent request.

By default Flask stores session data in a cookie, but in my case because I want to store larger amounts of data storing it in the cookie would add unnecessarily size to each HTTP request and response. To prevent that I’ll use a Flask extension named KVSession which replaces the normal Flask session mechanism with one that stores session data in a Redis database on my server.

In order to use KVSession in my app, I need to import the Flask Session dependency and the KVSessionExtension dependency. Since I want to use Redis as my data store I also need to import redis and the RedisStore dependency.

With the dependencies added, I need to specify a secret key that Flask uses to digitally sign the cookie that stores the session identifier, configure the RedisStore and then initialize the KVSessionExtension:

Now I can use the session object as I normally would in my Flask app, getting and saving key/value pairs which will be persisted to my Redis database:

KVSession takes care of setting a signed cookie with a unique Session ID for me, so all I need to do is return the TwiML response. On the next request from Twilio, KVSession will extract that ID and use it to repopulate the session object with the appropriate session values.

Application Initiated Conversations

So far I’ve talked entirely about scenarios where a user initiates a conversation with your application by sending an SMS to your Twilio phone. However, what happens if you want your application to initiate that conversation by sending an initial SMS message using Twilio REST API?

Your first instinct might be to try to use cookies like I demonstrated in the previous sections of this post. However, if you think about how cookies are passed between clients and servers, it quickly becomes clear that they will not work in this situation. Lets take a closer look at why.

In all of the examples I’ve shown so far Twilio serves as the HTTP client and your web application is the server. But when you use Twilios’ REST API, those roles are reversed.

Now your application is the HTTP client initiating HTTP requests to Twilios servers, and as we saw before, the client does not send a cookie on its initial request. Its only when the server returns a cookie that the client can then begin to include it in subsequent requests, and as you can see from the diagram above, the responses returned by Twilio when you make requests to the API do not include a cookie.

Wrapping It Up

Whether you need to facilitate a complex business process workflow, or just need a handy place to store some frequently used user data, using cookies in your Twilio application is easy. Because Twilio is a well-behaved HTTP client, all of the techniques you would normally use to leverage cookies in a website apply directly to using cookies with a Twilio application.

The tips to remember if you choose to use cookies in your own Twilio application are, Twilio will:

• create an association between a cookie and a unique To/From phone number pair,
• enforce a maximum cookie expiration time of four hours

Ready to get started tracking your own SMS conversations? Grab the source of the cookie example or the session example from GitHub and try it out yourself. And if you are using cookies in your own app, have questions about how Twilio sends and receives cookies or want to create your own SMS conversation tracking app, feel free to drop me a line via email or twitter.