Automated Survey with Node.js and Express

Time to read: 7 minutes

January 10, 2017
Written by
Kevin Whinnery
Twilion
Reviewed by
Paul Kamp
Twilion
Jose Oliveros
Contributor
Opinions expressed by Twilio contributors are their own
Kat King
Twilion
Orlando Hidalgo
Contributor
Opinions expressed by Twilio contributors are their own
Mica Swyers
Contributor
Opinions expressed by Twilio contributors are their own
Brianna DelValle
Twilion

automated-survey-node-express

Implementing your own automated survey with Twilio Voice and SMS can be a huge time saver when you need to collect feedback from a group of people. Whether it's participants in a social services program or a field service organization, you can quickly set up your own survey to collect structured data over the phone or via text message. Here's how it works at a high level:

  1. The end user calls or texts the survey phone number.
  2. Twilio gets the call or text and makes an HTTP request to your application for instructions on how to respond.
  3. Your web application serves up TwiML instructions to Gather or Record the user input over the phone, or prompts for text input with Message.
  4. After each question, Twilio makes another request to your server with the user's input, which your application stores in its database.
  5. Your application returns a TwiML response to Twilio with instructions to either ask the next question or end the survey.

What We Will Learn

This How-To demonstrates how to use TwiML to deliver a survey that can be completed via voice call. The survey actually works via SMS text messages, too, but we're going to focus on the looping logic necessary to conduct an interview over the phone. We will create a voice call flow using the Say, Record and Gather TwiML verbs.

You will also learn how to maintain conversation state in a database that spans multiple webhook requests. Beyond an automated survey, these techniques can be applied to implement more complex IVR systems or text message interfaces.

 

Editor: this is a migrated tutorial. Clone the original code from https://github.com/TwilioDevEd/survey-node/

{
  "name": "survey-node",
  "version": "1.0.0",
  "description": "an automated survey conducted via voice and sms",
  "main": "index.js",
  "scripts": {
    "start": "node index",
    "test": "node_modules/.bin/mocha ./**/*.spec.js --exit"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/TwilioDevEd/survey-node"
  },
  "keywords": [
    "twilio",
    "express",
    "survey"
  ],
  "author": "Kevin Whinnery",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/TwilioDevEd/survey-node/issues"
  },
  "homepage": "https://github.com/TwilioDevEd/survey-node",
  "dependencies": {
    "bluebird": "^3.5.5",
    "body-parser": "^1.19.0",
    "dotenv-safe": "^8.2.0",
    "express": "^4.17.1",
    "jade": "^1.11.0",
    "mongoose": "5.6.11",
    "morgan": "^1.9.1",
    "npm-check-updates": "^3.1.21",
    "tracer": "^1.0.1",
    "twilio": "3.33.4"
  },
  "devDependencies": {
    "chai": "^4.2.0",
    "lodash": "^4.17.15",
    "mocha": "^6.2.0",
    "supertest": "^4.0.2",
    "supertest-promised": "^1.0.0",
    "xmldom": "^0.1.27",
    "xpath": "0.0.27"
  }
}

About This Application

Like most Node.js web applications, this one relies on a number of smaller modules installed via npm to handle HTTP requests and store data. The key modules for this application are:

  • express - a popular web framework that helps us respond to HTTP requests to our application.
  • mongoose - an Object/Document Mapper (ODM) for MongoDB.
  • twilio - the Twilio Node module will help us generate TwiML responses to drive our interview.

Bootstrapping the Application

This is the Node file we will execute to serve up our web application. We load our app's configuration from an external file containing the HTTP port we want to run on and the MongoDB database connection string we need to store data using Mongoose.

We also define four routes to be handled by our web application. Three of the routes are webhooks that will be requested by Twilio when your Twilio survey number receives an incoming call or text, or when the results of a transcription job are ready. The fourth route will be used by our reporting UI to get the results of the survey from the database.

var app = require('./app');
var config = require('./config');
var http = require('http');

// Create HTTP server and mount Express app
var server = http.createServer(app);
server.listen(config.port, function() {
    console.log('Express server started on *:'+config.port);
});

Now that our application is all set up, let's look at the high level steps necessary to implement a voice interview via Twilio and TwiML.

The Voice Interview Loop

 The user can enter input for your survey over the phone using either their phone's keypad or by speaking. After each interaction Twilio will make an HTTP request to your web application with either the string of keys the user pressed or a URL to a recording of their voice input.

var VoiceResponse = require('twilio').twiml.VoiceResponse;
var SurveyResponse = require('../models/SurveyResponse');
var survey = require('../survey_data');

// Main interview loop
exports.interview = function(request, response) {
    var phone = request.body.From;
    var input = request.body.RecordingUrl || request.body.Digits;
    var twiml = new VoiceResponse();

    // helper to append a new "Say" verb with Polly.Amy voice
    function say(text) {
        twiml.say({ voice: 'Polly.Amy'}, text);
    }

    // respond with the current TwiML content
    function respond() {
        response.type('text/xml');
        response.send(twiml.toString());
    }

    // Find an in-progess survey if one exists, otherwise create one
    SurveyResponse.advanceSurvey({
        phone: phone,
        input: input,
        survey: survey
    }, function(err, surveyResponse, questionIndex) {
        var question = survey[questionIndex];

        if (err || !surveyResponse) {
            say('Terribly sorry, but an error has occurred. Goodbye.');
            return respond();
        }

        // If question is null, we're done!
        if (!question) {
            say('Thank you for taking this survey. Goodbye!');
            return respond();
        }

        // Add a greeting if this is the first question
        if (questionIndex === 0) {
            say('Thank you for taking our survey. Please listen carefully '
                + 'to the following questions.');
        }

        // Otherwise, ask the next question
        say(question.text);

        // Depending on the type of question, we either need to get input via
        // DTMF tones or recorded speech
        if (question.type === 'text') {
            say('Please record your response after the beep. '
                + 'Press any key to finish.');
            twiml.record({
                transcribe: true,
                transcribeCallback: '/voice/' + surveyResponse._id
                    + '/transcribe/' + questionIndex,
                maxLength: 60
            });
        } else if (question.type === 'boolean') {
            say('Press one for "yes", and any other key for "no".');
            twiml.gather({
                timeout: 10,
                numDigits: 1
            });
        } else {
            // Only other supported type is number
            say('Enter the number using the number keys on your telephone.'
                + ' Press star to finish.');
            twiml.gather({
                timeout: 10,
                finishOnKey: '*'
            });
        }

        // render TwiML response
        respond();
    });
};

// Transcripton callback - called by Twilio with transcript of recording
// Will update survey response outside the interview call flow
exports.transcription = function(request, response) {
    var responseId = request.params.responseId;
    var questionIndex = request.params.questionIndex;
    var transcript = request.body.TranscriptionText;

    SurveyResponse.findById(responseId, function(err, surveyResponse) {
        if (err || !surveyResponse ||
            !surveyResponse.responses[questionIndex])
            return response.status(500).end();

        // Update appropriate answer field
        surveyResponse.responses[questionIndex].answer = transcript;
        surveyResponse.markModified('responses');
        surveyResponse.save(function(err, doc) {
            return response.status(err ? 500 : 200).end();
        });
    });
};

It's up to our application to process and store the user's input, maintain the current state of the conversation, and respond back to the user. Let's dive into this flow to see how it actually works.

Responding to a Phone Call

To initiate the interview process, we need to configure one of our Twilio numbers to send our web application an HTTP request when we get an incoming call or text (remember, our app accepts text messages as well, even though we're focusing on the voice side).

Click on one of your numbers and configure Voice and Message URLs that point to your server. In our code, the routes are /voice and /message, respectively.

Automated Survey Webhook Setup

If you don't already have a server configured to use as your webhook, ngrok is a great tool for testing webhooks locally.

We've configured our webhooks in the Twilio Console. Next let's see how to respond to requests.

Responding to a Phone Call

The voice route maps to this Express handler function, which takes an HTTP request and HTTP response as arguments. From the request, we can access the phone number of the person calling in by the From POST parameter - we can use this to uniquely identify a person taking the survey.

We can also access the RecordingUrl, which contains any voice input from the user. Digits may also be present, which contains the string of keys entered by the user on their keypad. If this is the user's first call to our system or they failed to enter any input to the previous question, these values might be blank Strings.

We also create a TwimlResponse object that we will use to build up a string of XML we can ultimately render as a response to Twilio's request. It's not doing anything fancy - it just provides a JavaScript object that we can use to progressively assemble a valid TwiML string as our program executes.

var VoiceResponse = require('twilio').twiml.VoiceResponse;
var SurveyResponse = require('../models/SurveyResponse');
var survey = require('../survey_data');

// Main interview loop
exports.interview = function(request, response) {
    var phone = request.body.From;
    var input = request.body.RecordingUrl || request.body.Digits;
    var twiml = new VoiceResponse();

    // helper to append a new "Say" verb with Polly.Amy voice
    function say(text) {
        twiml.say({ voice: 'Polly.Amy'}, text);
    }

    // respond with the current TwiML content
    function respond() {
        response.type('text/xml');
        response.send(twiml.toString());
    }

    // Find an in-progess survey if one exists, otherwise create one
    SurveyResponse.advanceSurvey({
        phone: phone,
        input: input,
        survey: survey
    }, function(err, surveyResponse, questionIndex) {
        var question = survey[questionIndex];

        if (err || !surveyResponse) {
            say('Terribly sorry, but an error has occurred. Goodbye.');
            return respond();
        }

        // If question is null, we're done!
        if (!question) {
            say('Thank you for taking this survey. Goodbye!');
            return respond();
        }

        // Add a greeting if this is the first question
        if (questionIndex === 0) {
            say('Thank you for taking our survey. Please listen carefully '
                + 'to the following questions.');
        }

        // Otherwise, ask the next question
        say(question.text);

        // Depending on the type of question, we either need to get input via
        // DTMF tones or recorded speech
        if (question.type === 'text') {
            say('Please record your response after the beep. '
                + 'Press any key to finish.');
            twiml.record({
                transcribe: true,
                transcribeCallback: '/voice/' + surveyResponse._id
                    + '/transcribe/' + questionIndex,
                maxLength: 60
            });
        } else if (question.type === 'boolean') {
            say('Press one for "yes", and any other key for "no".');
            twiml.gather({
                timeout: 10,
                numDigits: 1
            });
        } else {
            // Only other supported type is number
            say('Enter the number using the number keys on your telephone.'
                + ' Press star to finish.');
            twiml.gather({
                timeout: 10,
                finishOnKey: '*'
            });
        }

        // render TwiML response
        respond();
    });
};

// Transcripton callback - called by Twilio with transcript of recording
// Will update survey response outside the interview call flow
exports.transcription = function(request, response) {
    var responseId = request.params.responseId;
    var questionIndex = request.params.questionIndex;
    var transcript = request.body.TranscriptionText;

    SurveyResponse.findById(responseId, function(err, surveyResponse) {
        if (err || !surveyResponse ||
            !surveyResponse.responses[questionIndex])
            return response.status(500).end();

        // Update appropriate answer field
        surveyResponse.responses[questionIndex].answer = transcript;
        surveyResponse.markModified('responses');
        surveyResponse.save(function(err, doc) {
            return response.status(err ? 500 : 200).end();
        });
    });
};

We've seen how to handle requests to our webhooks. Now lets go deeper into how to generate TwiML to redirect our users to the next question and generate speech from text.

Asking a Question

If either the user did not enter any input, it's the first question in the survey, or there's still another question after the current one, we will build a TwiML response that will ask the next question.

We define a few inner functions here to help us build our response. respond completes our TwiML response and sends XML content back to Twilio. say is shorthand for appending a string of text that will be read back to the user with Twilio TTS (text-to-speech) engine.

var VoiceResponse = require('twilio').twiml.VoiceResponse;
var SurveyResponse = require('../models/SurveyResponse');
var survey = require('../survey_data');

// Main interview loop
exports.interview = function(request, response) {
    var phone = request.body.From;
    var input = request.body.RecordingUrl || request.body.Digits;
    var twiml = new VoiceResponse();

    // helper to append a new "Say" verb with Polly.Amy voice
    function say(text) {
        twiml.say({ voice: 'Polly.Amy'}, text);
    }

    // respond with the current TwiML content
    function respond() {
        response.type('text/xml');
        response.send(twiml.toString());
    }

    // Find an in-progess survey if one exists, otherwise create one
    SurveyResponse.advanceSurvey({
        phone: phone,
        input: input,
        survey: survey
    }, function(err, surveyResponse, questionIndex) {
        var question = survey[questionIndex];

        if (err || !surveyResponse) {
            say('Terribly sorry, but an error has occurred. Goodbye.');
            return respond();
        }

        // If question is null, we're done!
        if (!question) {
            say('Thank you for taking this survey. Goodbye!');
            return respond();
        }

        // Add a greeting if this is the first question
        if (questionIndex === 0) {
            say('Thank you for taking our survey. Please listen carefully '
                + 'to the following questions.');
        }

        // Otherwise, ask the next question
        say(question.text);

        // Depending on the type of question, we either need to get input via
        // DTMF tones or recorded speech
        if (question.type === 'text') {
            say('Please record your response after the beep. '
                + 'Press any key to finish.');
            twiml.record({
                transcribe: true,
                transcribeCallback: '/voice/' + surveyResponse._id
                    + '/transcribe/' + questionIndex,
                maxLength: 60
            });
        } else if (question.type === 'boolean') {
            say('Press one for "yes", and any other key for "no".');
            twiml.gather({
                timeout: 10,
                numDigits: 1
            });
        } else {
            // Only other supported type is number
            say('Enter the number using the number keys on your telephone.'
                + ' Press star to finish.');
            twiml.gather({
                timeout: 10,
                finishOnKey: '*'
            });
        }

        // render TwiML response
        respond();
    });
};

// Transcripton callback - called by Twilio with transcript of recording
// Will update survey response outside the interview call flow
exports.transcription = function(request, response) {
    var responseId = request.params.responseId;
    var questionIndex = request.params.questionIndex;
    var transcript = request.body.TranscriptionText;

    SurveyResponse.findById(responseId, function(err, surveyResponse) {
        if (err || !surveyResponse ||
            !surveyResponse.responses[questionIndex])
            return response.status(500).end();

        // Update appropriate answer field
        surveyResponse.responses[questionIndex].answer = transcript;
        surveyResponse.markModified('responses');
        surveyResponse.save(function(err, doc) {
            return response.status(err ? 500 : 200).end();
        });
    });
};

Asking a Question

In our TwiML response, we need to include either a Gather tag or a Record tag to collect input from the user. Which tag we use depends on the question type in the survey.

In the Record use case, we also provide a transcription callback URL. Unlike Gather and Record, the transcription callback happens outside the loop of the call, sometime in the very near future (several seconds rather than minutes). So while you can't count on having the transcript results during the flow of the call, you can add the transcript to your database record to give the response a chance to help enrich that data a bit.

Another caveat here is that Twilio's transcription service is automated and not always super accurate. If transcription accuracy is critical for you, you might consider using a service with human translators like Rev.com.

var VoiceResponse = require('twilio').twiml.VoiceResponse;
var SurveyResponse = require('../models/SurveyResponse');
var survey = require('../survey_data');

// Main interview loop
exports.interview = function(request, response) {
    var phone = request.body.From;
    var input = request.body.RecordingUrl || request.body.Digits;
    var twiml = new VoiceResponse();

    // helper to append a new "Say" verb with Polly.Amy voice
    function say(text) {
        twiml.say({ voice: 'Polly.Amy'}, text);
    }

    // respond with the current TwiML content
    function respond() {
        response.type('text/xml');
        response.send(twiml.toString());
    }

    // Find an in-progess survey if one exists, otherwise create one
    SurveyResponse.advanceSurvey({
        phone: phone,
        input: input,
        survey: survey
    }, function(err, surveyResponse, questionIndex) {
        var question = survey[questionIndex];

        if (err || !surveyResponse) {
            say('Terribly sorry, but an error has occurred. Goodbye.');
            return respond();
        }

        // If question is null, we're done!
        if (!question) {
            say('Thank you for taking this survey. Goodbye!');
            return respond();
        }

        // Add a greeting if this is the first question
        if (questionIndex === 0) {
            say('Thank you for taking our survey. Please listen carefully '
                + 'to the following questions.');
        }

        // Otherwise, ask the next question
        say(question.text);

        // Depending on the type of question, we either need to get input via
        // DTMF tones or recorded speech
        if (question.type === 'text') {
            say('Please record your response after the beep. '
                + 'Press any key to finish.');
            twiml.record({
                transcribe: true,
                transcribeCallback: '/voice/' + surveyResponse._id
                    + '/transcribe/' + questionIndex,
                maxLength: 60
            });
        } else if (question.type === 'boolean') {
            say('Press one for "yes", and any other key for "no".');
            twiml.gather({
                timeout: 10,
                numDigits: 1
            });
        } else {
            // Only other supported type is number
            say('Enter the number using the number keys on your telephone.'
                + ' Press star to finish.');
            twiml.gather({
                timeout: 10,
                finishOnKey: '*'
            });
        }

        // render TwiML response
        respond();
    });
};

// Transcripton callback - called by Twilio with transcript of recording
// Will update survey response outside the interview call flow
exports.transcription = function(request, response) {
    var responseId = request.params.responseId;
    var questionIndex = request.params.questionIndex;
    var transcript = request.body.TranscriptionText;

    SurveyResponse.findById(responseId, function(err, surveyResponse) {
        if (err || !surveyResponse ||
            !surveyResponse.responses[questionIndex])
            return response.status(500).end();

        // Update appropriate answer field
        surveyResponse.responses[questionIndex].answer = transcript;
        surveyResponse.markModified('responses');
        surveyResponse.save(function(err, doc) {
            return response.status(err ? 500 : 200).end();
        });
    });
};

Now that we know how to ask questions and gather user input. Lets see how to store the survey state.

Updating Conversation State

Saving the user's response and maintaining the state of our conversation with the user is a concern best handled at the model layer of our server-side application, so we use our MongoDB-backed Mongoose model to handle this for us.

Abstracting the survey state from the controller also has the benefit of letting us re-use it for handling survey inputs from text messages. #winning!

var VoiceResponse = require('twilio').twiml.VoiceResponse;
var SurveyResponse = require('../models/SurveyResponse');
var survey = require('../survey_data');

// Main interview loop
exports.interview = function(request, response) {
    var phone = request.body.From;
    var input = request.body.RecordingUrl || request.body.Digits;
    var twiml = new VoiceResponse();

    // helper to append a new "Say" verb with Polly.Amy voice
    function say(text) {
        twiml.say({ voice: 'Polly.Amy'}, text);
    }

    // respond with the current TwiML content
    function respond() {
        response.type('text/xml');
        response.send(twiml.toString());
    }

    // Find an in-progess survey if one exists, otherwise create one
    SurveyResponse.advanceSurvey({
        phone: phone,
        input: input,
        survey: survey
    }, function(err, surveyResponse, questionIndex) {
        var question = survey[questionIndex];

        if (err || !surveyResponse) {
            say('Terribly sorry, but an error has occurred. Goodbye.');
            return respond();
        }

        // If question is null, we're done!
        if (!question) {
            say('Thank you for taking this survey. Goodbye!');
            return respond();
        }

        // Add a greeting if this is the first question
        if (questionIndex === 0) {
            say('Thank you for taking our survey. Please listen carefully '
                + 'to the following questions.');
        }

        // Otherwise, ask the next question
        say(question.text);

        // Depending on the type of question, we either need to get input via
        // DTMF tones or recorded speech
        if (question.type === 'text') {
            say('Please record your response after the beep. '
                + 'Press any key to finish.');
            twiml.record({
                transcribe: true,
                transcribeCallback: '/voice/' + surveyResponse._id
                    + '/transcribe/' + questionIndex,
                maxLength: 60
            });
        } else if (question.type === 'boolean') {
            say('Press one for "yes", and any other key for "no".');
            twiml.gather({
                timeout: 10,
                numDigits: 1
            });
        } else {
            // Only other supported type is number
            say('Enter the number using the number keys on your telephone.'
                + ' Press star to finish.');
            twiml.gather({
                timeout: 10,
                finishOnKey: '*'
            });
        }

        // render TwiML response
        respond();
    });
};

// Transcripton callback - called by Twilio with transcript of recording
// Will update survey response outside the interview call flow
exports.transcription = function(request, response) {
    var responseId = request.params.responseId;
    var questionIndex = request.params.questionIndex;
    var transcript = request.body.TranscriptionText;

    SurveyResponse.findById(responseId, function(err, surveyResponse) {
        if (err || !surveyResponse ||
            !surveyResponse.responses[questionIndex])
            return response.status(500).end();

        // Update appropriate answer field
        surveyResponse.responses[questionIndex].answer = transcript;
        surveyResponse.markModified('responses');
        surveyResponse.save(function(err, doc) {
            return response.status(err ? 500 : 200).end();
        });
    });
};

We have a general idea of how we want to persist survey state. Lets go into more details and have a look at what our schema looks like.

The Mongoose Schema

In our model, we create a schema that allows us to constrain and validate the form of documents that we insert into MongoDB. However, the Mixed type lets us store arbitrary JavaScript objects in the responses array, so we have some flexibility there with the objects we use to store our user's answers.

var mongoose = require('mongoose');
// Define survey response model schema
var SurveyResponseSchema = new mongoose.Schema({
    // phone number of participant
    phone: String,

    // status of the participant's current survey response
    complete: {
        type: Boolean,
        default: false
    },

    // record of answers
    responses: [mongoose.Schema.Types.Mixed]
});

// For the given phone number and survey, advance the survey to the next
// question
SurveyResponseSchema.statics.advanceSurvey = function(args, cb) {
    var surveyData = args.survey;
    var phone = args.phone;
    var input = args.input;
    var surveyResponse;

    // Find current incomplete survey
    SurveyResponse.findOne({
        phone: phone,
        complete: false
    }, function(err, doc) {
        surveyResponse = doc || new SurveyResponse({
            phone: phone
        });
        processInput();
    });

    // fill in any answer to the current question, and determine next question
    // to ask
    function processInput() {
        // If we have input, use it to answer the current question
        var responseLength = surveyResponse.responses.length
        var currentQuestion = surveyData[responseLength];

        // if there's a problem with the input, we can re-ask the same question
        function reask() {
            cb.call(surveyResponse, null, surveyResponse, responseLength);
        }

        // If we have no input, ask the current question again
        if (input === undefined) return reask();

        // Otherwise use the input to answer the current question
        var questionResponse = {};
        if (currentQuestion.type === 'boolean') {
            // Anything other than '1' or 'yes' is a false
            var isTrue = input === '1' || input.toLowerCase() === 'yes';
            questionResponse.answer = isTrue;
        } else if (currentQuestion.type === 'number') {
            // Try and cast to a Number
            var num = Number(input);
            if (isNaN(num)) {
                // don't update the survey response, return the same question
                return reask();
            } else {
                questionResponse.answer = num;
            }
        } else if (input.indexOf('http') === 0) {
            // input is a recording URL
            questionResponse.recordingUrl = input;
        } else {
            // otherwise store raw value
            questionResponse.answer = input;
        }

        // Save type from question
        questionResponse.type = currentQuestion.type;
        surveyResponse.responses.push(questionResponse);

        // If new responses length is the length of survey, mark as done
        if (surveyResponse.responses.length === surveyData.length) {
            surveyResponse.complete = true;
        }

        // Save response
        surveyResponse.save(function(err) {
            if (err) {
                reask();
            } else {
                cb.call(surveyResponse, err, surveyResponse, responseLength+1);
            }
        });
    }
};

// Export model
delete mongoose.models.SurveyResponse
delete mongoose.modelSchemas.SurveyResponse
var SurveyResponse = mongoose.model('SurveyResponse', SurveyResponseSchema);
module.exports = SurveyResponse;

Next, lets see how we will keep track of our data by using the type property on our models.

Storing Responses

Prior to saving a response from the user, we convert the raw string values submitted into data types that will be easier for us to work with as we analyze and visualize our survey data.

var mongoose = require('mongoose');
// Define survey response model schema
var SurveyResponseSchema = new mongoose.Schema({
    // phone number of participant
    phone: String,

    // status of the participant's current survey response
    complete: {
        type: Boolean,
        default: false
    },

    // record of answers
    responses: [mongoose.Schema.Types.Mixed]
});

// For the given phone number and survey, advance the survey to the next
// question
SurveyResponseSchema.statics.advanceSurvey = function(args, cb) {
    var surveyData = args.survey;
    var phone = args.phone;
    var input = args.input;
    var surveyResponse;

    // Find current incomplete survey
    SurveyResponse.findOne({
        phone: phone,
        complete: false
    }, function(err, doc) {
        surveyResponse = doc || new SurveyResponse({
            phone: phone
        });
        processInput();
    });

    // fill in any answer to the current question, and determine next question
    // to ask
    function processInput() {
        // If we have input, use it to answer the current question
        var responseLength = surveyResponse.responses.length
        var currentQuestion = surveyData[responseLength];

        // if there's a problem with the input, we can re-ask the same question
        function reask() {
            cb.call(surveyResponse, null, surveyResponse, responseLength);
        }

        // If we have no input, ask the current question again
        if (input === undefined) return reask();

        // Otherwise use the input to answer the current question
        var questionResponse = {};
        if (currentQuestion.type === 'boolean') {
            // Anything other than '1' or 'yes' is a false
            var isTrue = input === '1' || input.toLowerCase() === 'yes';
            questionResponse.answer = isTrue;
        } else if (currentQuestion.type === 'number') {
            // Try and cast to a Number
            var num = Number(input);
            if (isNaN(num)) {
                // don't update the survey response, return the same question
                return reask();
            } else {
                questionResponse.answer = num;
            }
        } else if (input.indexOf('http') === 0) {
            // input is a recording URL
            questionResponse.recordingUrl = input;
        } else {
            // otherwise store raw value
            questionResponse.answer = input;
        }

        // Save type from question
        questionResponse.type = currentQuestion.type;
        surveyResponse.responses.push(questionResponse);

        // If new responses length is the length of survey, mark as done
        if (surveyResponse.responses.length === surveyData.length) {
            surveyResponse.complete = true;
        }

        // Save response
        surveyResponse.save(function(err) {
            if (err) {
                reask();
            } else {
                cb.call(surveyResponse, err, surveyResponse, responseLength+1);
            }
        });
    }
};

// Export model
delete mongoose.models.SurveyResponse
delete mongoose.modelSchemas.SurveyResponse
var SurveyResponse = mongoose.model('SurveyResponse', SurveyResponseSchema);
module.exports = SurveyResponse;

We've seen how to persist answers to our questions. Next we'll see how guide the user to the next question.

Queueing Up the Next Question

After the current response is saved, we invoke the callback to our controller with the index of the next question in the survey. This may be longer than the actual length of the survey, which means we're finished asking questions!

var mongoose = require('mongoose');
// Define survey response model schema
var SurveyResponseSchema = new mongoose.Schema({
    // phone number of participant
    phone: String,

    // status of the participant's current survey response
    complete: {
        type: Boolean,
        default: false
    },

    // record of answers
    responses: [mongoose.Schema.Types.Mixed]
});

// For the given phone number and survey, advance the survey to the next
// question
SurveyResponseSchema.statics.advanceSurvey = function(args, cb) {
    var surveyData = args.survey;
    var phone = args.phone;
    var input = args.input;
    var surveyResponse;

    // Find current incomplete survey
    SurveyResponse.findOne({
        phone: phone,
        complete: false
    }, function(err, doc) {
        surveyResponse = doc || new SurveyResponse({
            phone: phone
        });
        processInput();
    });

    // fill in any answer to the current question, and determine next question
    // to ask
    function processInput() {
        // If we have input, use it to answer the current question
        var responseLength = surveyResponse.responses.length
        var currentQuestion = surveyData[responseLength];

        // if there's a problem with the input, we can re-ask the same question
        function reask() {
            cb.call(surveyResponse, null, surveyResponse, responseLength);
        }

        // If we have no input, ask the current question again
        if (input === undefined) return reask();

        // Otherwise use the input to answer the current question
        var questionResponse = {};
        if (currentQuestion.type === 'boolean') {
            // Anything other than '1' or 'yes' is a false
            var isTrue = input === '1' || input.toLowerCase() === 'yes';
            questionResponse.answer = isTrue;
        } else if (currentQuestion.type === 'number') {
            // Try and cast to a Number
            var num = Number(input);
            if (isNaN(num)) {
                // don't update the survey response, return the same question
                return reask();
            } else {
                questionResponse.answer = num;
            }
        } else if (input.indexOf('http') === 0) {
            // input is a recording URL
            questionResponse.recordingUrl = input;
        } else {
            // otherwise store raw value
            questionResponse.answer = input;
        }

        // Save type from question
        questionResponse.type = currentQuestion.type;
        surveyResponse.responses.push(questionResponse);

        // If new responses length is the length of survey, mark as done
        if (surveyResponse.responses.length === surveyData.length) {
            surveyResponse.complete = true;
        }

        // Save response
        surveyResponse.save(function(err) {
            if (err) {
                reask();
            } else {
                cb.call(surveyResponse, err, surveyResponse, responseLength+1);
            }
        });
    }
};

// Export model
delete mongoose.models.SurveyResponse
delete mongoose.modelSchemas.SurveyResponse
var SurveyResponse = mongoose.model('SurveyResponse', SurveyResponseSchema);
module.exports = SurveyResponse;

After you've finished asking questions, you might be interested in seeing how people responded to your survey. We won't spend too much time on visualizing the data, but we've included a bit of code in this sample app that shows you how you might approach that.

Checking Out The Results

In this app's static asset directory, you'll find an index.html file that contains some markup for displaying the results of our survey questions. It makes an Ajax request to our Express application to get the last 100 results from our survey to display here.

For phone survey responses, there is a link to listen to the recording in addition to the transcribed text sent to our application.

<!DOCTYPE html>
<html>
<head>
    <title>Automated Surveys</title>
    <link rel="shortcut icon" 
        href="//twilio.com/bundles/marketing/img/favicons/favicon.ico">
    <link rel="apple-touch-icon" 
        href="//twilio.com/bundles/marketing/img/favicons/favicon_57.png">
    <link rel="apple-touch-icon" sizes="72x72" 
        href="//twilio.com//bundles/marketing/img/favicons/favicon_72.png">
    <link rel="apple-touch-icon" sizes="114x114" 
        href="//twilio.com/bundles/marketing/img/favicons/favicon_114.png">

    <!-- CSS Dependencies -->
    <link rel="stylesheet"
        href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.2/css/bootstrap.min.css">
    <link rel="stylesheet" 
        href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">

    <!-- App CSS -->
    <link rel="stylesheet" href="main.css">
</head>
<body>

<!-- App Layout -->
<section id="main" class="container">
    <h1>Automated Survey Results</h1>
    <p>
        Want to display the results of your awesome Twilio survey? Here's
        how you might do that.
    </p>

    <h3>Showing <span id="total"></span> Total Responses</h3>
    <hr/>

    <div class="row">
        <div class="col-sm-6">
            <h3>"Please tell us your age."</h3>
            <canvas id="ageChart" width="300" height="300"></canvas>
        </div>
        <div class="col-sm-6">
            <h3>"Have you ever jump-kicked a lemur?"</h3>
            <canvas id="lemurChart" width="300" height="300"></canvas>
        </div>
    </div>

    <div class="row">
        <h3>"Who is your favorite Ninja Turtle and why?"</h3>
        <table id="turtleResponses">
            <tr>
                <th>Response Text / Transcript</th>
                <th>Recording</th>
            </tr>
        </table>
    </div>

</section>

<footer class="container">
    Made with <i class="fa fa-heart"></i> by your pals
    <a href="http://www.twilio.com">@twilio</a>
</footer>

<!-- JS Dependencies -->
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
<script src="chart.js"></script>

<!-- App script -->
<script src="index.js"></script>

</body>
</html>

Sometimes, the results aren't enough. So lets see how we can drill deeper and hear the actual recordings.

Playing Twilio Recordings

In the JavaScript that dynamically inserts the recording links into the page, we open the same RecordingUrl sent to us from Twilio in a new window. This works well enough, but you might consider playing them directly on the page with a more robust audio library like Buzz. When preparing the recordings to be played via HTML 5 audio, note that each recording is made available in either WAV or MP3 formats.

$(function() {

    // Chart ages
    function ages(results) {
        // Collect age results
        var data = {};
        for (var i = 0, l = results.length; i<l; i++) {
            var ageResponse = results[i].responses[0];
            var k = String(ageResponse.answer);
            if (!data[k]) data[k] = 1;
            else data[k]++;
        }

        // Assemble for graph
        var labels = Object.keys(data);
        var dataSet = [];
        for (var k in data)
            dataSet.push(data[k]);

        // Render chart
        var ctx = document.getElementById('ageChart').getContext('2d');
        var ageChart = new Chart(ctx).Bar({
            labels: labels,
            datasets: [
                {
                    label: 'Ages',
                    data: dataSet
                }
            ]
        });
    }

    // Chart yes/no responses to lemur question
    function lemurs(results) {
        // Collect lemur kicking results
        var yes = 0, no = 0;
        for (var i = 0, l = results.length; i<l; i++) {
            var lemurResponse = results[i].responses[1];
            lemurResponse.answer ? yes++ : no++;
        }

        var ctx = document.getElementById('lemurChart').getContext('2d');
        var ageChart = new Chart(ctx).Pie([
            { value: yes, label: 'Yes', color: 'green', highlight: 'gray' },
            { value: no, label: 'No', color: 'red', highlight: 'gray' }
        ]);
    }

    // poor man's html template for a response table row
    function row(response) {
        var tpl = '<tr><td>';
        tpl += response.answer || 'pending...' + '</td>';
        if (response.recordingUrl) {
            tpl += '<td><a target="_blank" href="'
                + response.recordingUrl 
                + '"><i class="fa fa-play"></i></a></td>';
        } else {
            tpl += '<td>N/A</td>';
        }
        tpl += '</tr>';
        return tpl;
    }

    // add text responses to a table
    function freeText(results) {
        var $responses = $('#turtleResponses');
        var content = '';
        for (var i = 0, l = results.length; i<l; i++) {
            var turtleResponse = results[i].responses[2];
            content += row(turtleResponse);
        }
        $responses.append(content);
    }

    // Load current results from server
    $.ajax({
        url: '/results',
        method: 'GET'
    }).done(function(data) {
        // Update charts and tables
        $('#total').html(data.results.length);
        lemurs(data.results);
        ages(data.results);
        freeText(data.results);
    }).fail(function(err) {
        console.log(err);
        alert('failed to load results data :(');
    });
});

That's All Folks

And that's it! You can reload the page on your localhost in your browser and watch as the results fly in from your users.

$(function() {

    // Chart ages
    function ages(results) {
        // Collect age results
        var data = {};
        for (var i = 0, l = results.length; i<l; i++) {
            var ageResponse = results[i].responses[0];
            var k = String(ageResponse.answer);
            if (!data[k]) data[k] = 1;
            else data[k]++;
        }

        // Assemble for graph
        var labels = Object.keys(data);
        var dataSet = [];
        for (var k in data)
            dataSet.push(data[k]);

        // Render chart
        var ctx = document.getElementById('ageChart').getContext('2d');
        var ageChart = new Chart(ctx).Bar({
            labels: labels,
            datasets: [
                {
                    label: 'Ages',
                    data: dataSet
                }
            ]
        });
    }

    // Chart yes/no responses to lemur question
    function lemurs(results) {
        // Collect lemur kicking results
        var yes = 0, no = 0;
        for (var i = 0, l = results.length; i<l; i++) {
            var lemurResponse = results[i].responses[1];
            lemurResponse.answer ? yes++ : no++;
        }

        var ctx = document.getElementById('lemurChart').getContext('2d');
        var ageChart = new Chart(ctx).Pie([
            { value: yes, label: 'Yes', color: 'green', highlight: 'gray' },
            { value: no, label: 'No', color: 'red', highlight: 'gray' }
        ]);
    }

    // poor man's html template for a response table row
    function row(response) {
        var tpl = '<tr><td>';
        tpl += response.answer || 'pending...' + '</td>';
        if (response.recordingUrl) {
            tpl += '<td><a target="_blank" href="'
                + response.recordingUrl 
                + '"><i class="fa fa-play"></i></a></td>';
        } else {
            tpl += '<td>N/A</td>';
        }
        tpl += '</tr>';
        return tpl;
    }

    // add text responses to a table
    function freeText(results) {
        var $responses = $('#turtleResponses');
        var content = '';
        for (var i = 0, l = results.length; i<l; i++) {
            var turtleResponse = results[i].responses[2];
            content += row(turtleResponse);
        }
        $responses.append(content);
    }

    // Load current results from server
    $.ajax({
        url: '/results',
        method: 'GET'
    }).done(function(data) {
        // Update charts and tables
        $('#total').html(data.results.length);
        lemurs(data.results);
        ages(data.results);
        freeText(data.results);
    }).fail(function(err) {
        console.log(err);
        alert('failed to load results data :(');
    });
});

Where to next?

If you're a Node.js developer working with Twilio, you might also enjoy these tutorials:

Click To Call

Learn how to use Twilio Client to convert web traffic into phone calls with the click of a button.

Two Factor Authentication

Learn to implement two-factor authentication (2FA) in your web app with Twilio-powered Authy.

Did this help?

Thanks for checking this tutorial out! If you have any feedback to share with us, we'd love to hear it. Connect with us on Twitter and let us know what you build!

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.