SMS and MMS Photo Filters with Twilio + Aviary

Time to read: 9 minutes

January 02, 2014
Written by
Kyle Kelly-Yahner
Twilion

Aviary

The following is a guest post by Ari Fuchs, Developer Evangelist at Aviary.

There are over 1.1 billion smart phone users in the world and 91% of them use their devices to take at least one photo per month. In 2015, more than 50% of all photos taken will be on smart phones.

In short, mobile photography is ubiquitous – and from its rapid growth has emerged a demand to capture, edit and share mobile photos anywhere, anytime and from any app.

Enter Aviary.

Aviary is a simple, yet powerful photo editing engine used by 70M people around the world every month. 20M use the “Photo Editor by Aviary” iOS or Android app and another 50M use it through the 6,000 mobile applications and websites it powers. The standard Aviary offering is a customizable widget, with a robust suite of tools including everything from simple crop and rotation to powerful features such as brightness, saturation, teeth whitening and redeye reduction to the ever-so-popular filters, stickers and frame overlays. Aviary’s mobile SDK’s are completely native, rendering all the edits locally on the device. The web widget is written in Javascript and the image rendering for images up to 1 megapixel happens right in the browser. For high resolution photo editing on the web, the widget communicates with the Aviary Render Farm to replay the user’s edits on the full resolution photo.

Note: High resolution editing is an add-on and incurs a monthly fee, but we’re offering a free tier for Twilio developers who want to try out this feature. Fill out this form and you’ll receive an email when your key has been enabled.

Now, you’re probably wondering how a communications API like Twilio can interface with a photo editor SDK like Aviary. To start, you’ve probably heard that Twilio has recently released support for MMS. That means that you can now text photos directly to a server endpoint. As I previously mentioned, Aviary has a server-side Render Farm for full resolution image editing on the web. Until recently, we’ve maintained this service exclusively to augment the web widget.

In this tutorial, I’ll show you how to bypass the Aviary web widget entirely, and how to communicate directly with our server-side rendering endpoint for editing photos programmatically. For the purposes of this tutorial, we’ll be creating a simple server endpoint that accepts a photo url via SMS, and responds with a url to an edited photo. We’ll then extend this app to accept and respond to MMS messages.

To start, let’s take a high level look at how the Aviary web widget communicates with the Rendering endpoint.

An Aviary Overview

As users edit their photos in the web widget, each time they ‘apply’ an operation it’s stored in a JSON “actionlist”. This actionlist can be exported using the getActionList() method on the Aviary web widget object (see full web documentation for more details). When the saveHiRes() method is called, this actionlist is posted to a server endpoint (/render) along with a url point at the high resolution photo that the developer had provided when launching the editor. The editor then polls a second endpoint (/render/{jobId}) until the photo has finished rendering. At this point, the onSaveHiRes(id, url) callback is called, and the developer has access to the full resolution photo.

The piece that you really care about here is getActionList(). You can use this to generate a pre-canned list of operations to apply to a photo.

Now let’s take a deeper look at those two endpoints, and see how we can communicate with them directly using the Aviary node.js wrapper.

Authentication

To start, every request to the Aviary server-side API must be accompanied by a signature that’s generated using your spikey, secret, a timestamp and a unique salt. Signature generation is fairly straightforward and the node.js wrapper abstracts it away for you entirely. You’ll still need to privde your apikey and secret, which you can get by creating an account at aviary.com/developers and creating a new web app. Your key will need to be whitelisted, so if you want to use this functionality in your own application, you’ll need to contact us at partners+twilio@aviary.com.

ActionList

Now that you have your whitelisted apikey, let’s jump right into editing that photo.
As I mentioned above in the high level Aviary description, every render operation requires an actionlist that has been generated by the editor. You can generate your own actionlists by launching the editor, completing a few operations in the UI, and programmatically calling the editor.getActionList() method. To make things a little easier, I’ve put together a list of actionlists that you can use in your app here.

There is one caveat. For any image operation that depends on a pixel position (ie, any of the draw tools, crop, etc), the server will do some simple calculations to scale that position for the full resolution photo. However, if the image that the actionlist is being applied to is a different aspect ratio that the image used to create the actionlist, the pixel position will either be incorrect, or not exist entirely. To avoid any issues, it’s recommended that you only create actionlists that apply operations to the entire photo. Those being any of the Effects (or filters), Autocorrect, or any of the slider tools, those being Brightness, Saturation, Contrast and Warmth.

Aviary Serverside API

Now that you have your actionlist, lets test it out with a simple node integration.

var Aviary = require('aviary').Aviary;

var key = {apiKey};
var secret = {apiSecret};
var url = {image url};
var actionlist = {stringified actionlist};

var aviaryClient = new Aviary(key, secret);
aviaryClient.renderAndWait(url, actionlist, function(err, newUrl) {
    console.log(newUrl);
});

First thing first, we require the Aviary NPM module and include the Aviary Constructor. When you instantiate the API wrapper, you pass in your activated Aviary api key and secret. The wrapper will take care of the signature generation for each request.

When you call aviaryClient.renderAndWait(), the wrapper makes the initial /render API call, which returns a <code>jobId. The method then polls the /render/{jobid} at 1 second intervals until a JobComplete status is returned along with a url pointing at the final rendered image – at which point, your callback is called with the new url.

Note, you can find some precanned action lists in the project repo here.

Tada! When you run this, you should see the rendered image url in your terminal.

Now let’s extend this with a simple server that can accept and respond to a Twilio SMS.

Twilio + Aviary App

If you’re reading this, you’re probably already familiar with handling Twilio SMS’s, but for the sake of completeness I’ll give you a quick refresher.

When you purchase a phone number from Twilio, you can set a Messaging Request URL. Any SMS message to this number will create a POST request to this url with the message in the POST body. When you receive this request, you can respond with a TwiML formatted response to send a SMS message back to the sender. Let’s take a look at what that looks like with a simple expressjs server.

Step 1: Create Express Server to Respond to SMS

var express = require('express'),
    app     = express(),
    twilio  = require('twilio');

var config = {
    authToken: process.env.TWILIO_AUTH_TOKEN
};

app.use(express.bodyParser());

app.post('/incoming', function(req, res) {
   // Validate Twilio request
  if (twilio.validateExpressRequest(req, config.authToken)) {
    var resp = new twilio.TwimlResponse();
    resp.message('Thanks for the message!')

    res.type('text/xml');
    return res.send(resp.toString());
  }
  else {
    return res.send('Nice try imposter.');
  }
});

var server = app.listen(process.env.PORT);
console.log("Listening on port %d in %s mode", server.address().port, app.settings.env);

The above code sample is pretty straightforward. First we create an express server and use the bodyParser() middleware to help us extract the SMS Body from the POST data. Next we create a POST handler for requests to the ‘/incoming’ endpoint. Before we do anything with the request, we verify that the POST actually came from Twilio using the handy validateExpressRequest method in the Twilio node.js library. Finally, we log the request Body to the console to verify that it’s working, and respond with an SMS message “Thanks for the message!”. You can read more about TwiML here, but for the sake of simplicity, we’ll be using the Twilio node.js library again to generate the appropriate response.

Ok! That wasn’t too tough, was it? Now let’s tie this together with the Aviary node.js module and get some photo filters in there.

Step 2: Detect Image Url and Render with Aviary

var express         = require('express'),
    app             = express(),
    twilio          = require('twilio'),
    Aviary          = require('aviary').Aviary,
    actionLists     = require('./actionlists.json'),
    aviaryClient    = new Aviary(process.env.AVIARY_API_KEY, process.env.AVIARY_API_SECRET);

var config = {
    authToken: process.env.TWILIO_AUTH_TOKEN
};

app.use(express.bodyParser());

app.post('/incoming', function(req, res) {

    // Validate Twilio request
    if (twilio.validateExpressRequest(req, config.authToken)) {
        var resp = new twilio.TwimlResponse();

        // Retrieve url from request body
        var urls = /(http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?/.exec(req.body.Body);
        if (urls && urls.length > 0) {
            var renderConfig = {
                url: urls[0],
                actionList: JSON.stringify(actionLists.filters.avenue)
            }

            // Call the Aviary API to render the image with a filter
            aviaryClient.renderAndWait(renderConfig, function(err, renderedUrl) {
                resp.message('Oooh pretty! ' + renderedUrl);

                res.type('text/xml');                
                return res.send(resp.toString());
            });
        } else {
            resp.message('Oops! Try sending a valid url.');

            res.type('text/xml');
            return res.send(resp.toString());
        }
    }
    else {
        return res.send('Nice try imposter.');
    }
});

var server = app.listen(process.env.PORT);
console.log("Listening on port %d in %s mode", server.address().port, app.settings.env);

First, let’s take a look at these two modules we’ve included.

var express         = require('express'),
   app             = express(),
   twilio          = require('twilio'),
   Aviary          = require('aviary').Aviary,
   actionLists     = require('./actionlists.json'),
   aviaryClient    = new Aviary(process.env.AVIARY_API_KEY, process.env.AVIARY_API_SECRET);

If you take a look at actionList.json, you’ll find a few pre-canned actionlists that were generated using the method described in the ActionList section above. We’ve also created an aviaryClient object that we’ll use to interact with the Aviary API.

Now let’s look at what we’ve done in the /incoming endpoint.

app.post('/incoming', function(req, res) {

   // Validate Twilio request
   if (twilio.validateExpressRequest(req, config.authToken)) {
       var resp = new twilio.TwimlResponse();

       // Retrieve url from request body
       var urls = /(http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?/.exec(req.body.Body);
       if (urls && urls.length > 0) {
           var renderConfig = {
               url: urls[0],
               actionList: JSON.stringify(actionLists.filters.avenue)
           }

           // Call the Aviary API to render the image with a filter
           aviaryClient.renderAndWait(renderConfig, function(err, renderedUrl) {
               resp.message('Oooh pretty! ' + renderedUrl);

               res.type('text/xml');               
               return res.send(resp.toString());
           });
       } else {
           resp.message('Oops! Try sending a valid url.');

           res.type('text/xml');
           return res.send(resp.toString());
       }
   }
   else {
       return res.send('Nice try imposter.');
   }
});

The first thing we do is extract any url’s from the SMS Body. Next, if a url is found, we call aviaryClient.renderAndWait() and pass in the supplied url, a pre-canned actionlist from the actionLists object, and a callback for when the render job is complete. When the render callback is called, we respond to the original SMS with a message that contains the rendered image url.

And that’s it! You’ve got a working node server that can accept an incoming SMS message containing an image url, and respond with an Aviary rendered image.

Well that was pretty cool, but you’ve got this awesome camera on your phone, and you’d probably like to be able to text a photo without without having to hosting it somewhere first. With an MMS supported Twilio number, you can do just that.

Step 3: Add MMS Support

Extending this app for MMS support is super simple. With Twilio’s new MMS support, certain Twilio phone numbers can now receive and send images along with text. Twilio makes our lives easier by hosting inbound images on their own CDN. If an inbound message POST contains any images, the NumMedia parameter in the message body will contain a value > 0, and the images can be retrieved with the MediaUrl# parameter (where # is the index of the media url). When you want to respond with an image in an outbound MMS (again, for supported numbers only), you can include a nested tag in the tag like so:

<!--?xml version="1.0" encoding="UTF-8"?-->
      Hello Jennyhttps://demo.twilio.com/owl.jpg

Let’s go ahead and extend our example to accept an image via an MMS message and send an MMS message back with the rendered image. Remember, you’ll need an MMS enabled number to make this work.

var express         = require('express'),
    app             = express(),
    twilio          = require('twilio'),
    Aviary          = require('aviary').Aviary,
    actionLists     = require('./actionlists.json'),
    aviaryClient    = new Aviary(process.env.AVIARY_API_KEY, process.env.AVIARY_API_SECRET);

var config = {
    authToken: process.env.TWILIO_AUTH_TOKEN,
    supportsMMS: false
};

app.use(express.bodyParser());

app.post('/incoming', function(req, res) {

    // Validate Twilio request
    if (twilio.validateExpressRequest(req, config.authToken)) {
        var resp = new twilio.TwimlResponse();
        var imageUrl;

        // Retrieve url from request 
        if (config.supportsMMS && req.body.NumMedia && req.body.NumMedia > 0) {
            imageUrl = req.body["MediaUrl" + 1];
        } else {
            var urls = /(http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?/.exec(req.body.Body);
            imageUrl = (urls && urls.length) ? urls[0] : null;
        }

        if (imageUrl) {
            var renderConfig = {
                url: imageUrl,
                actionList: JSON.stringify(actionLists.filters.avenue)
            };

            // Call the Aviary API to render the image with a filter
            aviaryClient.renderAndWait(renderConfig, function(err, renderedUrl) {

                // Generate the appropriate TwiML response
                if (config.supportsMMS) {
                    resp.message(function() {
                        this.body('Oooh pretty!');
                        this.media(renderedUrl);
                    });
                } else {
                    resp.message('Oooh pretty! ' + renderedUrl);
                }

                res.type('text/xml');                
                return res.send(resp.toString());
            });
        } else {
            resp.message('Oops! Try sending an image.');

            res.type('text/xml');
            return res.send(resp.toString());
        }
    } else {
        return res.send('Nice try imposter.');
    }
});

var server = app.listen(process.env.PORT);
console.log("Listening on port %d in %s mode", server.address().port, app.settings.env);

As you can see, instead of pulling the url out of the body text, we’re checking the NumMedia parameter first, and getting the url from MediaUrl1 when it’s available.

if (config.supportsMMS && req.body.NumMedia && req.body.NumMedia > 0) {
  imageUrl = req.body["MediaUrl" + 1];
} else {
  var urls = /(http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?/.exec(req.body.Body);
  imageUrl = (urls && urls.length) ? urls[0] : null;
}

Next, take a look at how we’re generating the TwiML response. Instead of just defining a message body, we’re also defining a node which contains the rendered url.

resp.message(function() {
  this.body('Oooh pretty!');
  this.media(renderedUrl);
});

Congratulations! You now have a working node app that can listen for and respond to both SMS and MMS messages.

One final note, in order to run your server, you’ll need to define some necessary environment variables:
export AVIARY_API_KEY=xxxxexport AVIARY_API_SECRET=yyyexport TWILIO_AUTH_TOKEN=zzz
You can get your Aviary key and secret by registering an account on http://www.aviary.com/developers and creating a web app. Fill out this form and someone from Aviary will whitelist your key.

You can find the full working example of this application on Github.

If you have any questions or comments, please tweet @arifuchs or email ari@aviary.com!

Photo courtesy of The Next Web

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.