Programmable Messaging - API Basics and Best Practices

Time to read: 8 minutes

June 18, 2019
Written by
David Lowes
Twilion

Twilio SMS Best Practices Series Part 1.png

Messaging can be complicated. The SMS ecosystem is comprised of mobile network operators (MNOs) and other application-to-person (A2P) network providers that sometimes route your message to its intended recipient, often over many hops. Delivery is not guaranteed, these players are known to have aggressive firewalls, spam algorithms and are protective of the traffic they send through their networks. On top of that, the backbone of SMS relies on old SS7 telephony signaling protocols for message transportation, developed back in 1975.

…. Sounds daunting already.

It’s not. Twilio’s Messaging API and underlying Super Network make it simple. Considering SMS is still the most ubiquitous, widely used messaging application in the world, it pays to get this right.

Sending Messages (one-way SMS)

Sending messages with Twilio is easy. Our SMS API allows you to send messages in your chosen programming language (check out our Helper Libraries). Here are the key concepts and “gotchas” -

  • Authenticate yourself. HTTP requests to the SMS API are protected with HTTP Basic Authentication. The username is your Twilio Account SID and the password is your Auth Token. You can find both of these pieces of data in your Twilio Console.
  • In addition to those values, there are 3 parameters you will always need when making a request to the Message API - to, from and body. Here is sample code: 
SMS cURL request parameters.png
  • The to and from phone numbers must be E.164 formatted.
  • A single message may be broken up into many segments, which may change the way the message is formatted and incur additional costs. This is explained more below.

Receiving Messages (two-way)

Your use-case may require that your customers send messages back to you. Two-way SMS with Twilio is not complicated. Here are the key concepts and “gotchas” -

  • Webhooks. Inside your Twilio console, within the phone number settings, you must define the URL that we should HTTP POST the content to.
  • This URL can point to a service within your infrastructure, or a standalone location, such as a TwiML Bin. This decision will depend on your use-case. TwiML Bins are great for basic message flows, but not ideal for logging and conversational message threads.
  • Regardless of location, you will need to use TwiML, our XML Markup language, to handle the message response. You could do this with the help of one of our libraries, or just use TwiML. I have illustrated both below (there are advantages to using a helper library, such as allowing you to use conditional statements in your code for more control of how to handle the message response): 
    <?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Message><Body>This is what TwiML looks like!</Body></Message>
</Response>
    const MessagingResponse = require('twilio').twiml.MessagingResponse;
const app = express();

app.post('/sms', (req, res) => {
    const twiml = new MessagingResponse();

    twiml.message("This is what TwiML looks like!");

    res.writeHead(200, { 'Content-Type': 'text/xml' });
    res.end(twiml.toString());
});

Message Logs (retrieving and deleting)

Need logs? No problems. We make a range of messaging data available to you. Here are the key concepts and “gotchas” -

  • Use the same resource to GET your message. If you sent a message using
    POST https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages.json     
    and then received this response from your request: 
SMS API Response

You would GET the message using the same URL, adding that unique SID - 
GET https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages/{sid}.json
  • If you wanted to GET all your messages, you would just use /Messages, This might be a significant query, which is why we have additional parameters that you can use in your query, such as date_sent, allowing you to filter messages in a specific time period.
  • Twilio stores message logs in compliance with strict data policies. Whilst your data is stored indefinitely, it's accessable to only you. Twilio implements technical and organizational security controls as appropriate to the risk associated with that data. Each piece of SMS data is marked either PII or NOT PII. Fields marked as PII are also marked with an MTL - a Minimum Time to Live. This is the number of days after creation that data will be stored in Twilio's systems for carrier reconciliation, tax management, or other business purpose that requires us to hold the data. Outside of MTL, data can be deleted from Twilio storage immediately. To see what data properties are tagged as PII and MTL, checlk out the API Message Parameters page.
  • You can always DELETE message logs before this time using the same resource:
    DELETE https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages.json

Delivery Status (DLRs, statusCallbacks & Feedback Resource)

Importantly, you need to know whether your messages are being delivered to your customers.

The best way of understanding your message deliverability is to subscribe to Status Callbacks. These callbacks allow you to receive receipts on the changing state of a message as it’s transported from Twilio, through operators, to the destination recipient. There are currently 9 potential states a message might be in - Accepted, Queued, Sending, Sent, Receiving, Received, Delivered, Undelivered, Failed. The states Receiving and Received, are for inbound messages, which means that someone has replied to a message and your Twilio number has received it. All other states are for outbound messages. To register for these events, you must specify a Callback URL using the statusCallback parameter, as shown below - 

statusCallback URL

This will allow you to receive asynchronous state change notifications and ultimately keep track of message deliverability.

How do you know my message is being delivered, you may ask?

Twilio receives Delivery Receipts (DLRs) from the operators we pass your message to, who are themselves notified by the physical device that a message was received. Although this sounds simple, device limitations, carrier/operator infrastructure and the sheer volume of messages sent through networks make this complex, and DLRs are sometimes unreliable. Twilio does its best with the information made available to us to provide you with accurate data. If you feel that your Delivered events are not right, speak to us.

Another way for savvy customers to receive data on message delivery is to use our MessageFeedback subresource. This is suitable for use-cases that require some trackable action, such as URL clicks and temporary passwords. In this case, delivery is tracked because the intended action was taken, so it involves some manual lifting at your end. This resource simply gives you another layer of deliverability tracking.

Throughput and Message Queues

Throughput often gets confused with API Rate Limits, which is separate (but related) and explained in detail below.

Throughput is simply the rate in which Twilio sends, or more accurately, dequeues each message. This is measured per phone number, per second. Different number types have different throughput limitations. I explain the individual phone number types in more detail in Part 3 of this blog series, but for now, here is a good list of number types and their Messages Per Second (MPS) limits.

If your use-case calls for a significant volume of messages to be sent over a short time window, we recommend that you use Messaging Services. The next blog post in this series is dedicated to this product.

But let’s say that you don’t use a Messaging Service and you only have a single phone number which is sending messages at a rate >1mps, or two numbers sending messages at a rate >2mps (or three at a rate >3mps… you see where I’m going with this), then your messages are going to get queued. Using simple math; if you make 100 API requests in a single second (which might be the maximum you can make, but maybe not - read below) and you are using a single Long code phone number to send those messages, which has a limit of 1MPS, then it’s going to take… surprise surprise, 100 seconds to send all of those messages as each message is dequeued on the second until the last message is dispatched and there are no more messages in the queue.

Throughput and MPS restrictions are set to maximize message deliverability and protect you from carrier bans.

API Rate Limits

Even without throughput restriction, you can only send a limited amount of messages per second, because Twilio imposes an account level API rate limit. Each Twilio account can make concurrent GET requests and non-GET (including POST) requests at any point in time.

To understand more about Twilio rate limits, reference these articles:

As highlighted above, a crucial “gotcha” with Twilio API Limits is that they are enforced at the account level, not at a service level, or phone number level. So keep this in mind if you are running other services, such as Voice or Lookups.

Message Segments

SMS messages are actually made up of 140 byte chunks segments. This is helpful to know considering Twilio charges by the segment. If you write long messages, then you may actually be sending, 2, 3 or 4 segments. Each Segment is 140 bytes..

But how do I know how many bytes I’m using? And more importantly, how many characters make up 140 bytes? Well that depends on character encoding.

140byte segment sms twilio

Encoding

The GSM-7 character set is the standard alphabet for SMS, written up in GSM 03.38 and always supported on GSM networks. GSM-7, as the name suggests, are 7 bit characters. There are 8 bits in a byte… Uh oh, more math:

(140 bytes*8 bits)/7 bits = 160

Okay, so there are 160 characters in a segment?! Yes and no.

Encoding your message in GSM-7 is only possible if you use the GSM-7 character set. So be cautious of what characters you are using. Many text editors do not support GSM-7 encoding. If you use any non-GSM-7 characters in your message, twilio will fallback to using UCS-2 encoding for the whole message. These characters are not 7 bits, they are 16 bits, which significantly reduces the amount of characters you can use in each message:

(140 bytes*8 bits)/16 bits = 70

Well at least we now know that our message segments are either going to be 160 characters or 70 characters, depending on the characters I use. Ermmm, not quite...

Data Header and Concatenation

If you send one SMS that is in fact, 4 segments, we must make sure that your customer receives that message in the order in which it was written. That is, segment 1 first, segment 2 second, segment 3 third and segment 4 last. To do this, we must instruct the receiving device to assemble the whole message in the order which it was sent.

User Data Header Twilio

We do this by prepending a User Data Header to the message, containing that specific concatenation information. This takes up 6 bytes (48 bits). 

Character Limits

So how many characters can you actually send in an SMS message until it is broken up into a new segment?

If you strictly use the GSM-7 character set -

  • 140 bytes in a segment
  • Minus 6 bytes for the User Data Header = 134 bytes
  • Converted to bits = 134*8 = 1072 bits
  • GSM characters = 7 bits
  • 1072 / 7 = 153 characters

If you use any character that is not in the GSM-7 character set -

  • 140 bytes in a segment
  • Minus 6 bytes for the User Data Header = 134 bytes
  • Converted to bits = 134*8 = 1072 bits
  • UCS-2 characters = 16 bits
  • 1072 / 16 = 67 characters

Troubleshooting Error Messages

To end Part 1 of this Twilio SMS Best Practices series, I would like to direct you to our Error and Warning Dictionary. If your API Request to Twilio is complete and valid, we will most commonly send you a 200 OK response and create a log of that request. If your request is invalid, we will throw a 2XXXX error code.

We may also give you an error code if your API request is valid but something else has gone wrong and we cannot deliver your message. A Failed or Undelivered message gerneally generally returns a 30XXX error code.

SMS Best Practice Guide

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.