How to Connect Twilio to your Web Application

How to Connect Twilio to your Web Application

When you are building with Twilio, a common and flexible approach to defining the behaviour of incoming calls and messages is by using webhooks . When your Twilio phone number receives a message or a call, Twilio will bundle up all the relevant data about the incoming messages and calls into HTTP requests which it then makes to the webhook URL that you provide.

This is a nice boundary because there is a lot of flexibility in how you build a web service – you can use almost any programming language. By contrast, there is little flexibility on the connectivity side – however you decide to handle the webhook requests, it is completely necessary to have a publicly-accessible URL for Twilio to call. You have free choice for how to make that happen, but if you're just getting started with Twilio or with coding you might not feel sure where to host your code.

How you find the answer depends on a few things:

• what phase your project is in - prototyping with you as the only user? Deploying for a community project, or for an app with hundreds of users that needs to run unattended for a long time?
• how familiar you are with Cloud technologies and networking
• your needs with specific hardware - do you need to control or monitor some physical system such as in home automation.

If you have doubts about the options you have for hosting, this post is for you.

Before we start, I will point out that you can find a few different ways to handle incoming calls and messages without leaving the Twilio platform using Twilio Functions or Studio or TwiML Bins . These all use webhooks under the hood, but for this post I'm considering the common situation that you are writing and hosting your own app to handle the webhooks , outside of Twilio.

Why Can't Twilio Reach My Laptop?

Before diving in let's address the core issue: why can't you direct Twilio to your laptop? Despite having fast download speeds and enjoying low latency gaming, it's highly unlikely that you have a direct internet connection at home or work. Typically, you'll be connecting through an ISP with a dynamic IP and Network Address Translation ( NAT ) to make outgoing network connections to the internet.

NAT lets you use the Internet from multiple devices using a single Internet connection, for example your home router is typically a NAT device. By setting a NAT device as your computer's gateway , every packet that needs to get to the Internet has its "Source" address overwritten with the NAT's public address. When responses come back from the Internet they are therefore routed to the NAT device which again rewrites – this time the "Destination" address - to get back across your wi-fi to your computer.

This is sometimes called masquerading, and to keep track of it all your NAT device maintains a "Translation Table" of source/destination address-and-port pairs which need rewriting in both directions. Entries in the translation table typically have a timeout which can be set as low as 30s

NAT is a vital part of how the Internet works at the scale we have today. In particular, the ability to use NAT to masquerade for several devices has helped mitigate the problems of IPv4 address exhaustion (the other big part of that is IPv6 adoption, but that is a topic for another post which I won't go into here). You should also note here that NAT does not only apply to HTTP traffic, but all network traffic between your computers and the internet.

You can now see why NAT is a two-sided coin:

• Pros: you can connect all your devices to the internet through a single connection.

• Cons: You cannot establish a connection from the Internet to a specific device on your network unless it is in the NAT translation table, which is normally caused by an outgoing connection between the same hosts (and on the same ports) and in any case will quickly time out.

In other words, there simply isn't an address that Twilio can use for webhook requests that will get through to your dev machine. The NAT is in the way. So you have to do a little work. What options are there for hosting Twilio webhooks? This post will discuss 5 ways.

1. SSH Tunnels and Tunneling Applications

2. Self hosting

3. Web Hosting as a Service

4. VPS

5. VPN

SSH Tunnels and Tunneling Applications

You might have used SSH as a tool to remotely access a remote server, but it can do a lot more than that. One of its advanced functions is called "tunneling": the ability to use the encrypted connection that `ssh` creates to listen and forward traffic for another application. This article covers ssh tunneling in more detail – all we need is the mental model that if your app is on your laptop and you can ssh to a Virtual Private Server (like a Cloud Instance) then tunneling can make it appear that your service is running on the VPS itself, like a little helpful ghost that, similar to NAT, rewrites source/destinations to forward packets over the tunnel. You've essentially created another, private NAT with your laptop hard-coded into the translation table.

If you have a VPS then you can set this all up yourself, however you might need to configure ssh on the host and in any case the tunnel will only work as long as your ssh connection stays up. NAT will want to tidy up this connection if it's idle for too long, for which we have SSH Keepalive options. Overall, manually setting this up is not a very satisfying solution – you need a VPS and some linux sysadmin knowhow too, and it depends on you (ab)using the NAT Translation Table to keep packets flowing.

Luckily there are services which offer a similar model, relieving you of the burden of configuring and maintaining the VPS, and ensuring that the tunnel remains up.  Several options have been collected and written up on the awesome-tunneling repo on GitHub. I have used ngrok a lot but there are many alternatives, offering extra features like automatic https certificates and servers in different countries. They can be extremely quick and convenient, but of course your app is still running on your dev machine and that is not ideal when you want to use that computer for something else or turn it off and go outside.

Tunneling App Summary

Pros

• easy to set up and reconfigure

• several free options to get started

• only needs your dev machine

Cons

• your dev machine and the tunnel need to remain up and running for your Twilio app to work

• it also depends on your home internet connection

• usage or rate limits might apply

Self Hosting

On some ISPs it is possible to configure your router to forward incoming connections to any computer on the home network, by port. This depends on being able to configure port forwarding in your router, which is essentially like adding a permanent entry to your NAT Translation Table. Although your ISP might change your IP address from time to time, you can use a Dynamic DNS service to keep a stable hostname. You can expect some downtime when the IP address changes in this case, and if that's unacceptable it may also be possible to buy a static IP address from your ISP. You are taking the first steps to turn your home into a data center.

It's worth noting that this isn't even possible with all ISPs. My ISP runs Carrier Grade NAT which means that the ISP runs its own (huge) NAT Translation Table which I cannot possibly modify in this way.  To know if your ISP uses CG-NAT, check the public IP address of your router (the address which your ISP has assigned to it). If that is in one of the private address spaces then it's very likely CG-NAT. 

Self Hosting Summary

Pros

• hosting at home means your Twilio app can control home servers. Home automation or pet-monitoring apps would need this.

• you will learn a lot about IP networking, routing and DNS. Perhaps more than you want to.

Cons

• dependent on your home compute infrastructure and internet connectivity

• might be against your ISP's terms of service (please check first!)

• self-hosting opens you up to DDoS and other attacks, at your home

Code Hosting as a Service

Cloud companies offer a lot of hosting options and although they differ a lot in implementation and use, it is appropriate to group Functions/Platform/Containers-as-a-Service together for this article (often known as FaaS, PaaS, and CaaS respectively). Each of them lets you upload some code to a cloud provider, and the provider takes care of deploying and running it. This includes services like AWS Lambda and Azure Functions (FaaS), Google App Engine and Heroku (PaaS), and Azure Container Instances or EKS (CaaS).

You might have to configure endpoint URLs and other parts of the platform. A single server handling Twilio webhooks is typically not a very heavy load and I tend to choose small instance sizes, all the better if they're in a free tier.

Code Hosting Summary

Pros

• your provider will take care of  hardware, deployment, or maintenance as part of the service

• useful integration with other cloud services (if you use them)

• deployment from source-control is usually supported which lets you work with others more easily

Cons

• you may save time using a vendor-specific integration but this can lock you into that ecosystem

• not every programming language and runtime gets the same level of support

• "cold start" times for some platforms may cause you to reach the webhook timeout

If you are a Javascript dev, you should consider Twilio Functions. You can upload Javascript code which can be used as a webhook handler with automatic verification of the X-Twilio-Signature header so that you know only Twilio can call your code.

Virtual Private Servers

A Virtual Private Server (aka Cloud Instance, or just VPS) is a more flexible, and a more complex option. I already talked about using one as a host for SSH tunneling, but this time you're actually copying your code to the VPS and running it there. You'll choose an Operating System and instance type, launch the instance, configure a network with security rules, set up SSH access, log in, apply updates, install whatever application server and libraries you need, add your code, acquire a domain name, configure a static IP address and DNS, open the right ports on the firewall, get an HTTPS certificate from letsencrypt, set a calendar reminder to check for security updates and refresh certificates regularly, and… "as simple as that", you have your webhook app running with a public URL.

VPS Summary

Pros

• good practise to learn about cloud and Linux sysadmin

• if you want to integrate with other Linux services or use an esoteric programming language or web hosting app, this will definitely be flexible enough

Cons

• defining, creating, securing and maintaining a virtual server on someone else's infrastructure is complicated

• resource limits, security updates, upgrades and failover are all your responsibility to manage

Virtual Private Networking

The last option is the most flexible, and perhaps predictably the most complicated to set up. You might use a VPN at work or college and it feels like some inscrutable thing sitting on your computer doing some kind of magic with the networking. But this is Clarke's Third Law in action: VPN systems are surprisingly uncomplicated in theory. If you followed the section on NAT above then VPNs are only a couple of steps further (again, that's for another post).  However the practical implementation is often not easy.

A VPN can connect two computers (or networks of computers) to make them appear as one private network, using encryption to safely cross whatever public networks are in between. In this example I am suggesting a VPN between your app server behind NAT and a VPS on the Internet. This would give you a secure connection between the VPS which has the public IP address, and your app server which is under your full control and can be anywhere with an Internet connection. VPN software can connect through a NAT as in this example, and can also be used to join computers which are both behind different NATs using STUN and TURN protocols. It is not possible to establish a VPN with Twilio but managing your own VPN on computers that you control using something like Wireguard  is very possible, with effort.

If you find yourself thinking that a VPN might be the solution you need, I would encourage you to check out managed services like ExpressVPN or Tailscale before starting out on the self-hosting journey. It's interesting and instructive to figure out how these things work, but it might be some time before you can get back to working on your Twilio app.

VPN Summary

Pro

• you will be a networking expert by the time you finish setting up your VPN

• having a VPN makes a lot of other networking problems easier as you can treat it exactly like a LAN and let the VPN software worry about things like NAT traversal and keepalives.

Con

• highly complex to set up

• monitoring and maintenance all fall to you as well

• configuration errors can end in network isolation which can be difficult to fix quickly

Conclusion

Ultimately, your choice depends on your specific needs, the maturity and scale of your application, where you are in your journey with Twilio, how much time you want to spend on networking side-quests, and which platforms you are most comfortable with. My recommendations are:

• For developers getting started or looking for simplicity and ease of use, try a managed tunneling tool.

• For teams working on collaborative projects or solo devs looking to keep code running while they're not building, code hosting platforms offer ease of use with integration and workflow benefits. If you're using Javascript then Twilio Functions is built for you.

• For projects which need access to specific hardware, VPNs provide an additional layer of protection and flexibility at the cost of configuration complexity.

All of these methods have been used successfully to build many apps on Twilio - and plenty more options exist besides. These are my recommendations for how to proceed if you're not sure, based on my experience which is surely different from yours. I can't wait to hear how you build on Twilio.

Matthew Gilliard, Developer Evangelist, Twilio