Delay API calls to Twilio with Rails, Active Job and Sidekiq

Delay API calls to Twilio with Rails, Active Job and Sidekiq

Performance is key in web applications. Snappy websites make for better user experiences, higher conversion rates and better user retention. A swift application response causes less stress on servers trying to respond to many users too. There are many ways to improve the performance of a web application in Rails and I want to look at one of those today.

Performing long running, blocking tasks during the course of a request is a top way to slow down responses for any web application. I’m talking things like sending emails, generating PDF or CSV files or making HTTP requests to 3rd party APIs. All of these things take a relatively long time compared to other actions normally performed during the course of a request. If you’re using Twilio within an application the last point might stick out at you.

The best thing to do with long running tasks like this is move them away from the request itself and perform them in the background. This allows your application server to respond to requests swiftly and get the job done without affecting performance for your site’s users or tying up web server processes. Since version 4.2 Rails has included Active Job, a library which makes it easy to delay long running tasks and perform them in a background queue. In this post we will explore how we can Active Job to speed up our application.

Let’s queue!

In order to show how we can speed up an application’s response times by queueing tasks to happen in the background, we’re going to need an application to test against. Rather than build one up right now, we’re going to use one of the example applications available in the Twilio tutorials. Let’s take a look at Click to Call, a simple form on your website that takes a user’s phone number and calls them back. You can build the application up using the tutorial or grab the application from GitHub.

What you’ll need

To complete this you’ll need a few things:

• A Twilio account (sign up for a free Twilio account if you don’t have one already)
• A Twilio number that can make phone calls
• A way to tunnel from a public URL to your localhost (I really like ngrok)
• Ruby and bundler installed
• Redis (check out the quick start guide for redis to get it up and running)

To run the application, grab your Twilio credentials from your account dashboard and the phone number, then:

1. Clone the application: $ git clone https://github.com/TwilioDevEd/clicktocall-rails
2. cd to the application: $ cd clicktocall-rails
3. Install the gems: $ bundle install
4. Set your environment variables:
   

• Migrate the database (there’s no migrations for now, but this keeps the application happy): $ bundle exec rake db:migrate
• Run the tests: $ bundle exec rake test
• Run the server: $ bundle exec rails server
• In a separate console tab, set up your tunnel $ ngrok http 3000

• Now we’re ready to test this application. Open up the application in your browser using the public URL from your tunnel. You’ll see a form asking for your phone number. Enter your number and submit the form, you’ll see a success message and then receive a phone call.

This all looks great, right? “What’s Phil’s problem?”, you might be asking. Let’s take a look at the logs:

That’s right, loading up the home page took just 38ms and starting the call took just over a second. Whilst this is reasonable when you are starting up a project or not dealing with a lot of traffic, over time this is going to hurt.

Kill the server with Ctrl-C and let’s sort this out.

Speeding it up

Like I said in my introduction, the solution to requests that contain long running, blocking tasks like this is to move the task into the background and perform it outside of the web request. We’re going to do that now using Active Job. First, let’s take a look at the whole action we’re dealing with:

The action receives a phone number in the params and creates a ContactTwilio::REST::Client object, which we then use to create a call from our Twilio number to the entered phone number, pointing it at the connect_url (see the connect action for what happens here).

I’ve highlighted the API call in the code above. This is our long running task that we need to move out to a background job. So, let’s create a job to handle that. On the command line, enter:

This gives us two new files, app/jobs/make_call_job.rb and test/jobs/make_call_job_test.rb. To ensure our job does the right thing, we can copy parts of the tests from the controller test to the job test.

If we run the tests now, we’ll see a failure.

So let’s fix it. Thankfully all the code we need is already in the controller as I showed earlier. So we need to update the MakeCallJob. Active Job classes need only define two things, the queue name, which we can leave as :default, and a perform method. We can move the API request from our controller over to the perform method in our job.

Run the tests again and they will pass. Good stuff, we’re halfway through. Now the controller needs changing too. We need to ensure that instead of placing the call using the API, the action will add a job to the queue with the right arguments. We can do this with the assert_enqueued_with assertion. Let’s open up test/controllers/twilio_controller_test.rb and change the following test:

In order to use this fancy assert_enqueued_with test assertion, we need to include the Active Job test helpers. At the top of the test, insert one extra line:

Run the tests again and we’ll see another failure.

We need to update the controller to queue the job instead of placing the call. Active Job classes define an instance method called perform but to place jobs on the queue we the class method perform_later. By default perform_later will place the job on the queue to be processed as soon as there is a worker available. You can also schedule jobs to run at a set time in the future using the set method followed by perform_later, like MakeCallJob.set(wait_until: 3.days.from_now).perform_later.

As we want this job to run as soon as possible, let’s update the controller to call MakeCallJob.perform_later with the two arguments we defined for the job; the number to call and the URL for Twilio to request when the call is answered. Open up app/controllers/twilio_controller.rb and replace the lines where the Twilio::REST::Client is created and used with that call to the job class.

Run the tests again and you’ll see success. Let’s test this out for real now. Start up the server again, enter your phone number and wait for the call.

The call comes through correctly. But the log is still showing more than a second to process this. What is happening?

Choosing a queue

Active Job is great because it gives us a very simple interface to queue up jobs. However, out of the box Active Job only supplies a default implementation that immediately executes jobs inline. In order to move the work into the background we need to supply a different back end. Active Job supports a number of popular Ruby job queues including Sidekiq, Resque and Delayed Job. There’s a list of all the adapters available in the Active Job documentation as well as their features.

I like Sidekiq, it uses Redis as a store for the jobs (Delayed Job integrates well with Active Record, but this can be slow) and thread based workers (Resque uses process based workers, which can take up a lot more memory). We’ll complete this example using Sidekiq, but I encourage you to read up on the options and choose the one that is right for your application.

Open up the Gemfile and add Sidekiq as a dependency. Add the line beneath gem 'twilio-ruby':

Run $ bundle install to install Sidekiq and then open up config/application.rb to set the Active Job backend:

Now all we need to do is restart our application, start up the Sidekiq process, which will load up some workers to process our jobs, and watch the application speed along. Make sure you have Redis running at this point, Sidekiq will need it. If you have installed Redis, you can start it in a separate console tab with the command:

To start Sidekiq, open up a new console tab, cd to the application directory, set the same environment variables as the application tab, using your credentials and Twilio number, and start Sidekiq:

Sweet ASCII art, right? Load up your app again, enter your phone number and wait for the call. You should see little difference in the usage of the application, but check out the application logs.

That’s right, our call action now only takes ~50ms to run. Using Active Job and Sidekiq we have taken all the load off our web application processes and left it for the background process to deal with.

Further considerations

There are a few things that need to be considered when you move work like this from the web process to the background. Like we discussed earlier, the queue backend that you choose is important. For example, if you’re not already using Redis, then adding Resque or Sidekiq will mean adding another dependency for your project. If you need to process a lot of jobs, then using Active Record as part of Delayed Job might not be appropriate either.

There’s the user interface to think of too. In this example if the call creation fails the user won’t see any error, they just won’t get a call. It might be a good idea to track the job progress and update users later with any errors.

Join the queue

In this post we took a controller action that contained a long running task and sped it up its response by moving the work to the background. We saw how Active Job makes it easy to do this within Rails apps and we chose Sidekiq as the backend queue system. Check out the final code here on GitHub.

Now you’ve tried out Sidekiq, it could be time to consider what the other queue backends can do for you. Resque and Delayed Job are the other popular ones, but check out Sneakers, which uses RabbitMQ to store jobs, or Que, which relies on PostgreSQL.

If you want to do the same for sending SMS messages check out the textris gem which wraps up SMS into an Action Mailer like interface, including easy Active Job integration.

Active Job is one of those Rails features that took a long time to arrive, but makes life easier now it’s here. If you’ve got any questions about using background jobs or interesting ways you use queues, drop them here in the comments or hit me up on Twitter at @philnash.