Handle Twilio Debugger Events with Python and Flask

Handle Twilio Debugger Events with Python and Flask

Twilio offers a wide range of APIs. One Twilio account can run many applications, and perhaps your team develops and maintains a specific range of them. To best debug your applications when (and not if) there are errors, it would be nice to filter through debugger events and only get the notifications that you need to see.

In this post, I’ll show you how to leverage Twilio’s Debugger Alert Triggers to receive, process and store the error events from your selected Twilio application. That is, you’ll be able to write an application that listens to events from the Twilio Debugger and writes relevant content about them into a comma separated values (CSV) file that you maintain on your machine. In a more advanced project, you could write this into a cloud database instead. The idea is the same.

Tutorial requirements

We will be using Python, so I’ll assume you have Python 3.6+ installed. In addition, we’ll need

• Flask: to create a web application that Twilio can send Debugger events to.
• ngrok: to allow the local Flask application to be available on the Internet. If you haven’t yet, download ngrok!
• A Twilio account. If you are new to Twilio, you can create a free Trial account. Please review the Trial account’s limitations.
• A Twilio number that your account owns with voice capabilities. If you are on a Trial account with credit, you won’t be charged.
• A phone that you can use to make calls to your Twilio number and try the app. (If your Twilio account is on Trial, this number will need to be verified.)
• The Twilio Helper Library for Python: to validate incoming requests.

How Twilio Handles Application Errors

No one ever wants to hit errors, but they will certainly happen! In order to troubleshoot errors, and debug your code while developing with Twilio, you can take advantage of Debugger events. In this case, we will create some errors on purpose in order to learn better how Twilio’s Debugger behaves, and how to handle the events.

While Debugger events can come from any Twilio product, in this case I will use Programmable Voice, that is, phone calls.

We’ll first need a Twilio number with voice capabilities in our account, and then we’ll make some test calls to it, from our personal phone. When the call comes in the Twilio number, the error will occur.

First, check if you own a Twilio number that you could use to make calls to and test this app. So, navigate to your incoming phone numbers page in Console.

If you own a Twilio number with voice capabilities that you are happy with, perfect. But if you want another Twilio number, then you can search for and purchase a new number from Console. Note that trial accounts can own up to one Twilio number, so you may need to release the number you own, before being able to purchase a new one.

Once you are happy with the Twilio number, navigate to its configuration page by clicking on the number, and here we will create the root cause of our sought for error.

When a call comes in to your Twilio number, Twilio will request the URL or application that you have configured in the “A call comes in” field. We can now pass a URL that returns something that Twilio doesn’t expect. Let’s use, for example, https://something.unexpected.com/.

When you click on Save, you will have configured the number!

Now, dial this number from your mobile phone or a phone that you can make calls from. You will immediately hear an error message: “We’re sorry, an application error has occurred, goodbye!” and the call will then end. While in Twilio’s console, you will also see the Debugger icon in flashing red notifying you of an event in the Debugger’s console:

If you click on the event, you’ll find a lot more details about that specific error, including the Request Inspector logs from the error at the application level.

The documentation for Error 12100 is pretty useful, it tells you that Twilio wasn’t able to obtain valid TwiML in the response from the webhook request. That’s an application error!

As you noticed, it takes a good few clicks and navigating various pages in Twilio’s console to find details about the error. Also, Debugger only stores events for 30 days. If you want to keep a long time record of your events, you need to save them elsewhere. In the sections below, I’ll show the steps to create an application that will smartly log Debugger events in a CSV file, avoiding the need to log in to the console to view them.

Debugger’s Alert Triggers

By contrast with the screenshot above showing only one error event, a typical Twilio account with a few projects in production will have many events in Debugger, coming from different applications that you own, and products that you use. Debugger can be rather noisy! There are some tools that Twilio offers to help organize your errors:

• You can receive email notifications when an error occurs. You can customize this to a certain point. But you will need to log in to Debugger’s console to inspect each notification. Having to log in to Twilio’s console and navigate to Debugger every time you want to view error notifications may not be your ideal experience as a developer. Perhaps your production application builds on a specific Twilio product, phone number, or webhook, and you just would like to parse through the many events from Debugger and see only the events that you need in order to best maintain your project.
• In addition to email notifications, Twilio offers a Monitor API. This allows you to fetch existing Debugger events up to 30 days old. By design, this doesn’t provide real time alerts.

In the following section, I will discuss a third option that Debugger offers: Alerts Trigger.  I will show you the steps to have a Python application written with the Flask framework that offers the following relevant features:

1. Listen to real-time Debugger events
2. Save event details in a CSV file on your local file system
3. Control over what events to log and for how long

Let’s build this!

Write a Flask application to log Debugger events

I’ll walk us through the steps to get this application up and running, but if you are new to Flask and web-applications in Python, I’d recommend you visit the Flask quickstart. Also, for a full-blown tutorial on Flask, see Miguel Grinberg’s mega tutorial.

Create a Python Virtual Environment

Let’s follow Python’s best practices and create a separate directory for our Debugger Flask application, including a virtual environment. This is where we’ll install our dependencies:

In the last step, I used Python’s package installer pip3 to install theFlask framework, which will allow us to create the web application.

Listening to Debugger events with Flask

Let’s create an application that will listen for requests from Debugger and will print all the information received when an error event is created. To test it out, we will run the application locally, then open a tunnel with ngrok, and instruct Twilio’s Debugger to forward errors to ngrok’s URL. We’ll also trigger an error on purpose like we did before.

My code for the application to listen to Debugger alerts is shown below.

Save this code in a file with extension .py, such as basicDebugger.py and run it from your terminal. You’ll see something like this:

This is the Flask app! It is now running, and listening for requests on port 5000 in your localhost. When the endpoint http://127.0.0.1:5000/DebuggerPayload or http://localhost:5000/DebuggerPayload receives a POST request, the application will run the code in the debugger() function.

That code, instructs to gather the payload from the POST request, organize it into a dictionary and print it in the terminal. In addition, it responds to the client with 200 OK, which is how we tell Twilio that our request completed successfully.

The problem is that Twilio cannot reach this endpoint running locally on your computer, so we’ll need to open a tunnel from our local network to the public Internet. This is where ngrok enters the picture. It’s a really powerful tool, and the free account is very useful in testing environments. Leave the Flask application running, and start ngrok in a new terminal as follows:

Now you’ll see something like this:

This means that you can now use the URL https://<code>.ngrok.io/DebuggerPayload in Twilio’s Debugger Alert Triggers, and any request sent to this URL will be forwarded by ngrok directly into our Flask application. What comes before .ngrok.io is a hashed domain that ngrok provides you with. This part of the URL will change every time you run ngrok http 5000 in the terminal.

Now that we have the simple debugger endpoint and ngrok both running and awaiting connections, let’s tell Twilio to talk to us when an event in Debugger is created. Navigate to the Webhook & Email Triggers page in Console, enter the https:// ngrok forwarding URL with /DebuggerPayload appended to it and save.

When you pass a URL like in the image above, Twilio will make a POST request with the following payload every time an event is published in Debugger:

• Sid: Unique identifier of this Debugger event (has the form: NOxxxx).
• AccountSid: Account SID where the Notification SID belongs (if it is from a sub-account)
• ParentAccountSid: Parent account for AccountSID. This parameter only exists if the above account SID is a sub-account SID.
• Timestamp: timestamp for the event being issued, in ISO 8601 format.
• Level: Severity of the Debugger event. Possible values are Error and Warning.
• PayloadType: application/json
• Payload: JSON data specific to the Debugger Event. This is the main piece of information!

Let’s generate a test error now.

Remember from the previous section that when a call comes in to your Twilio number, Twilio will request the (obviously invalid) URL https://something.unexpected.com/, and consequently an error will occur.

Make a call from your phone to the Twilio number, and hear the error message. Then, in the terminal where you are running the Flask application, you’ll see the dictionary with the JSON payload, printed by the debugger() function. It will not look nice and tidy as the cleaned up version below, but the output will have the following information:

Let’s discuss this payload.

When the call came in to the Twilio number, Twilio made a request to the webhook https://something.unexpected.com/ as this is what we had configured to handle the incoming phone call. In the response to this webhook request, Twilio expects a TwiML payload with instructions on what to do with the incoming call. Since the webhook URL didn’t respond with valid TwiML, an error notification is published in Debugger. The event from Debugger is uniquely identified by the Notification SID, which starts with NO.

Notice the nested keys and values in the JSON payload. By inspecting this structure we can code our application to log what we think is more relevant to debug our application in the future.

Let’s write a new route for a second endpoint in the basicDebugger.py file. This will be a bit more specific than the debugger listener application we had in /DebuggerPayload.

In this new endpoint we’ll have two new features:

• We will only consider error events coming specifically from our URL https://something.unexpected.com/ and ignore any others. This is going to serve as an example of how to filter error events.
• And, the new endpoint /SimpleDebugger will add a line in a local CSV file with the payload! The library I’m using for this is csv, from the Python Standard Library.

Below is the updated code. If you are still running the previous version of the Flask server, stop it with Ctrl-C, then save this new version and run it. Once you have the new version running, you will need to instruct Twilio to now POST to this new endpoint. You can do so by removing /DebuggerPayload and replacing it with /SimpleDebugger in the webhook for Webhook & Email Triggers page in Console.

With this endpoint if the error is encountered when Twilio hits a different webhook, Debugger will POST to your debugger application, but your application will ignore that event. Your application will only write in the CSV file when the URL that was involved is https://something.unexpected.com/. That is what the code above says. You could easily amend this code to add more conditions on what events to consider.

Once an error from https://something.unexpected.com/ occurs, a new line in the file csv_debugger.csv is written:

Other kinds of errors will produce other error codes and messages (the last two columns). Twilio publicly documents all the errors that it produces with possible causes and possible solutions.

From the above application, you can now decide what to log to best debug your applications!

Bonus: Secure your Debugger application

Since the ngrok URL is openly accessible on the Internet, it is genuinely available to any clients who may make requests to your application. So far, nothing suggests in the code that this is an application for Twilio’s Debugger. In fact, if you want to make a POST request with cURL, Postman or other HTTP clients, your application will respond 200 OK and maybe even write in the CSV file, if you carefully construct a payload to include a dictionary with the keys that we are logging.

To safely hear events only from your Twilio Debugger, we can validate the origin of the requests. That is, Twilio provides details in the request so that your server-application can confirm that the client is really Twilio. This is done with the request signature found in the “X-Twilio-Signature” header, which Twilio adds in every request that it makes. Using as input both your Twilio account’s AuthToken and the payload of the request, the validator object from the Twilio Helper Library will produce a signature, and if it matches the one that actually comes in the request’s header, that will be solid evidence that the client here is your Twilio account and not an impostor.

You can install the Twilio Helper library in the virtual environment by running the following command in your terminal:

With Twilio’s Python Helper Library we can use the validator object and verify the signature on our side and reject all invalid requests.

You can find the Auth Token required to do signature validation in the Twilio Console. Copy your Auth Token and then set it in an environment variable, either in your prompt, in your .bash_profile file or where it makes sense for your shell. For bash type shells, the command is:

Now, let’s add a /SafeDebugger route to the basicDebugger.py file.

After you save the new version of the Flask application and restart it, navigate to Debugger in Console to instruct Twilio to now POST events to the new /SafeDebugger route.

And that’s it! You now have an application that only writes events coming genuinely from your Twilio account, and only logs events in your local file system when the application that encountered an error is our relevant one:  https://something.unexpected.com/.

Conclusion

In this blogpost, we discussed the basic functionality of the Twilio Debugger’s Alert Triggers. We wrote an application that updates a CSV file in our local system with relevant payloads from errors occurring in our Twilio account from one specific webhook.

With the basic idea outlined above, I hope this will inspire you to build a Debugger application for your production webhooks and applications. This can be used for anything in Twilio: TwiML Bins, Studio Flows, Taskrouter callbacks, etc.

You could add more conditions on when and what to write in the CSV file. Moreover, you could code conditions to write in different CSV files depending on what the URL the error came from, or for all URLs, but depending on the Error number. After this, you will be able to have a nice dataset with plenty of information about your Twilio errors.

I’m a member of the Personalized Support team in Twilio, and my email is nsznajderhaus [at] twilio [dot] com. Let me know if the content of this blog was useful, or if you have any questions or feedback! I’d love to hear from you and I can’t wait to see what you build!