Outsmart Voice Mail with Human Detection Using Keypress Prompts

Time to read: 10 minutes

September 18, 2020

Written by

Alan Klein

Twilion

Reviewed by

Tilde Thurium

Twilion

Zack Pitts

Twilion

Samuel Eddy

Twilion

Paul Kamp

Twilion

Outsmart Voice Mail with Human Detection Using Keypress Prompts

One of the challenges of programmatically connecting parties over the phone is accounting for voice mail. From the perspective of the underlying telephony network, a human answering a call looks identical to an automated answering machine.

Twilio provides Answering Machine Detection (AMD) that can be enabled when placing an outbound-api call via the /Calls resource. However, when connecting to a party using the Studio Connect Call To widget (or TwiML Dial), we must use an alternative approach called "Human Detection".

Human Detection requires the answering party to positively acknowledge the call by pressing any button – something voicemail cannot do. In this post, we will use Human Detection, to make sure voicemail remains on Twilio rather than inadvertently being left on the end-users device.

Where might you need Human Detection?

In my last post, Build Voice Mail with SMS and Email Notifications Using Twilio Studio, Functions, and SendGrid, we used the metadata DialCallStatus returned by the Studio Connect Call To Widget to intelligently route unanswered or busy calls to Twilio voicemail and alert via Email or SMS of our voicemail.

What about calls that are answered by the dialed devices voicemail? That’s where human detection comes into play.

By detecting that you are connected to a human, you can trigger other flows and Functions in a communications flow.

Prerequisites

In order to follow this tutorial, you will need:

A Twilio account and active number. If you don’t have an account, go ahead and create one now. (If you sign up and upgrade with this link you'll also get another $10 credit)
If you want to have the transcribed voicemails sent to you via email, you’ll also need a Twilio SendGrid account, with a verified sender.

SendGrid Account

In order to use the SendGrid API to send emails we need a SendGrid API key. Create a SendGrid account (if you don't have one already), confirm your email address, then head to the API Keys area in the SendGrid dashboard and create an API Key with “Full Access”.

Be sure to save the API Key Secret to your notepad.

Verified Sender

You also need to verify one of your email addresses to use it with SendGrid. To do this, go to the Senders page under Marketing, click “Create New Sender”, fill out the information, then go to that email inbox and click the link in the verification email.

Build our Twilio Functions

Now, let's write three Twilio Functions.

The first Function we write will initialize a SendGrid mail client and be used to email you a transcribed voicemail recording and URL.

If you have visited Twilio Functions before, things may look a little different. Twilio recently released a new Functions and Assets UI! It is a fantastic upgrade and you can read more about it here. We now have quite a number of practical Twilio Function examples on this same page, which you can modify to meet your needs for other projects.

Configure

To make requests with SendGrid's API, we need a way to authenticate. To do this, we'll use your SendGrid API Secret.

Open up your Twilio console and head to the Functions configure section. If you followed the previous blog post, we created a Function service called, "OutsmartingVoiceMail", and the first two Twilio Functions for Email and SMS notifications from that blog are the same as this blog. If you didn't follow the previous blog, no worries, create a new service as detailed below.

Add a new environment variable called SENDGRID_API_SECRET and paste in your API Key Secret (begins with SG).

Open up your Twilio console and head to the Functions overview section.
Under Overview, click the blue “Create Service” button.
Give it the service name, “OutsmartingVoiceMail”
Under Settings -> Environment Variables, add a new environment variable called SENDGRID_API_SECRET, paste in your API Key Secret (begins with SG) you saved from the step above, then click Add.

We also need to create environment variables for the email addresses you’ll be sending to and from.

Create a TO_EMAIL_ADDRESS variable, enter the email to receive your voicemails, then click “Add”.
Create the variable FROM_EMAIL_ADDRESS, enter one of the verified sender emails in your SendGrid account, then click Add. It’s best to use two of your personal emails when testing. Emails from a company domain (e.g. @twilio.com) will be blocked if not verified properly.

In addition, to the Environment Variables, we need to include the @sendgrid/mail library in our Dependencies, under Settings, Dependencies.

Add @sendgrid/mail and the version 7.2.5, then click “Add”.

Write the Functions

Forward Voicemail via Email

In the forwarding to email function, we’ll set up our voicemail to forward a link and transcription when a caller leaves a new recording.

At the top left of the screen, click the blue “Add” + button and select “Add Function”. You can leave the Function at the default, “Protected”. Protected means that only requests from your account with the proper X-Twilio-Signature can successfully execute the Function.
Add a name and path for your Function, I used /forwardVMViaEmail

The Function is included below. Feel free to copy, paste, and save. If you'd like to understand what's happening, read on.

//Initialize SendGrid Mail Client
const sgMail = require('@sendgrid/mail');

// Define Handler function required for all Twilio Functions
exports.handler = function(context, event, callback) {

// Build SG mail request
sgMail.setApiKey(context.SENDGRID_API_SECRET);
       // Define message params
       const msg = {
         to: context.TO_EMAIL_ADDRESS,
         from: context.FROM_EMAIL_ADDRESS,
         html: `<p><strong>New Voicemail from:</strong> ${event.From}</p>
               <strong>Transcription is:</strong><br>${event.TranscriptionText}<p><a href=${event.RecordingUrl}>
               Click here to Listen to the Recording</a></p>`,
         subject: `New Voicemail from: ${event.From}`,
       };
       // Send message
       sgMail.send(msg)
       .then(response => {
           console.log("Neat.")
           return callback();
       })
       .catch(err => {
           console.log("Not neat.")
           return callback(err);
       });
};

How your Function works

There’s a bunch of code there, and it might not be immediately obvious what we’re doing. Here’s how it works.

At the top of your Function, we initialize the @sendgrid/mail module which will help us craft a mail request to SendGrid.
We then call Twilio's handler function, which is required for all Twilio Functions. For more information about how the handler function works and what the context, event, and callback arguments are used for, check out the docs here.
Next, we build the SendGrid request following the guidelines in the docs here.
Since Twilio Functions are written in JavaScript, we’ll use the SendGrid Node.js library.
The first thing we do when building our request is use the setApiKey() method which will authenticate our request. Inside it, we place our API Secret, which we pull from the Function context.
Next, we build our message parameters. Again, we pull our to and from emails from context, and we also reference a couple variables, url and From, from the event argument of our Function.
Finally, we call the send() method to send the request to SendGrid for delivery. Inside the promise handler, we print a message to let us know the request was accepted, then we embed our empty Function return callback() which is required in every Twilio Function. We add a catch statement so we can know what’s wrong if SendGrid fails our request.

Forward Voicemail via SMS

In the forwarding voicemail to SMS function, we’ll send an SMS with a link when you receive a voicemail recording.

At the top left of the screen, click the blue “Add” + button and select “Add Function”. You can leave the Function at the default, Protected.
Add a name and path for your Function, I used /forwardVMViaSMS.

The Function is included below. Feel free and copy, paste. Edit the to and twilioFrom (the Twilio phone number the SMS will come from) to meet your unique configuration, then save. If you'd like to understand what's happening, read on.

// Description
// Send a single SMS

exports.handler = function (context, event, callback) {
   // Make sure under Functions Settings tab:
   // "Add my Twilio Credentials (ACCOUNT_SID) and (AUTH_TOKEN) to ENV" is CHECKED
    const twilioClient = context.getTwilioClient();
    // Begin Necessary Configuration
   let to = '+14075550100';
   let twilioFrom = '+15095550100'
   // End Necessary Configuration
  
   let body = `New Voicemail from: ${event.From}\n\nTranscription is:\n${event.TranscriptionText}\n\nListen to the recording\n${event.RecordingUrl}`;
    twilioClient.messages
     .create({
       body: body,
       to: to,
       from: twilioFrom,
     })
     .then((message) => {
       console.log('SMS successfully sent');
       console.log(message.sid);
       return callback(null, 'success');
     })
     .catch((error) => {
       console.log(error);
       return callback(error);
     });
 };

How your Function works

This Function initializes the Twilio Node helper library so we can call the /Messages resource to send an outbound SMS.
We pass in the from, which is a Twilio number in our account, to send the SMS from.
We pass in the to, which is the mobile number we will send the SMS to.
We pass in the body, which will contain our transcription and a link to listen to the recording. Note, email is optimal for long voicemail transcriptions as a single SMS can have a maximum size on most carriers of 1,600 bytes (usually 10 segments). An SMS is charged per segment, you can get more details on what a segment is here, Programmable SMS - API Basics and Best Practices.
The .then and .catch are called based on the resulting success or failure.

Human Detection

Finally, we need to get to the heart of the matter and detect when a human answered the phone. The following code (and code comments) explain how we’re performing this function.

At the top left of the screen, click the blue Add (‘+’) button and select “Add Function”. You can again leave the Function at the default, Protected.
Add a name and path for your Function, I used /humanDetection
The Function is included below. Update studioFlowSid and dialedPartyNumber to your unique settings.

The studioFlowSid starts with FW and will be found once we create our Studio flow in the next section, for now leave the placeholder and we will come back to this. The dialedPartyNumber is the person receiving the call.
Feel free to copy, paste, and save the code. Click the blue “Deploy All” button to deploy your functions.
If you'd like to understand what's happening, read through the comments in the code.

// Description
// Twilio Studio "Human Detection" logic, which requires the dialed party to press any key
// To answer the call. This avoids connecting the calling party with the called parties personal
// voicemail. Currently, Studio does not natively support the <Number> url parameter.
// https://www.twilio.com/docs/voice/twiml/number#attributes-url

// This Function performs multiple roles using URL query parameters to determine the proper response
// If reason=humanDetection, return TwiML to collect dialed parties response, w/Gather action URL of disposition
// If reason=disposition, it means the dialed party pressed a key to accept the call (they are not VM)
// If reason=dialStatus, the actionURL of the TwiML Dial is providing the DialCallStatus/DialCallDuration
// to determine if the call should return to the Studio Flow to leave a voicemail or to hang up the call
// since the dialed party was answered by a human (so no need to leave a VM)

// Can Twilio tell whether a call was answered by a human or machine?
// Alternatives to AMD
// https://support.twilio.com/hc/en-us/articles/223132567-Can-Twilio-tell-whether-a-call-was-answered-by-a-human-or-machine-

exports.handler = function(context, event, callback) {
    // Begin Necessary Configuration
   const studioFlowSid = "FWf28e..." // placeholder until we create flow
   const dialedPartyNumber = "+14075550100"
   // End necessary configuration
  
   const accountSid = context.ACCOUNT_SID
   const studioWebhookURLReturn = `https://webhooks.twilio.com/v1/Accounts/${accountSid}/Flows/${studioFlowSid}?FlowEvent=return`;
  
     const twiml = new Twilio.twiml.VoiceResponse();
   const functionPath = `https://${context.DOMAIN_NAME}${context.PATH}`;
    if (event.reason === "humanDetection") {
     let gather = twiml.gather({
       action: `${functionPath}?reason=disposition`, input: 'dtmf',
       timeout: 5,
       numDigits: 1
     })
     .say("Press any key to answer this call");
     twiml.hangup();
     return callback(null, twiml);
   }
  
   if (event.reason === "disposition") {
       twiml.say("Connecting call now");
       return callback(null, twiml);
     }
    
   if (event.reason === "dialStatus") {
     if (event.DialCallStatus === "no-answer" || event.DialCallStatus === "busy" || (event.DialCallStatus === "completed" && !event.DialCallDuration )) {
       twiml.redirect(studioWebhookURLReturn)
       return callback(null, twiml);
     } else {
       twiml.hangup();
       return callback(null, twiml);
     }
   }
  
   const dial = twiml.dial({ action: `${functionPath}?reason=dialStatus` });
   dial.number({ url: `${functionPath}?reason=humanDetection` }, dialedPartyNumber);
    return callback(null, twiml);
 };

Studio Flow

Instead of walking through the creation of the Studio Flow, you should import my JSON representation of the flow which you can find here. I’ll explain the flow in turn.

Import the Studio Flow

1. To get started, go to the Manage Flows Studio page, then click either the red “Create new Flow” button (if this is your first Flow) or the red plus (‘+’) sign (if it’s not your first Flow). !Create new Flow button when it's your first flow Or !Create new Flow button when you have existing flows 1. Next, give your Flow a name. I called mine `studioLeaveVoicemailBlogWhisper` – either way, click “Next”. Scroll down and select “Import from JSON” from the menu, and click “Next” again. 1. Remove the initial `{}` and replace the content of the edit box with the JSON that can be downloaded here, and click “Next”. You should see the Studio canvas rendered out for you, as below. !Flow overview with Human Detection 1. Customize the widgets in the imported flow for your unique environment. - Edit the “initialGreeting” widget with your custom message and click “Save”. - Edit the “redirectDialHumanDetection”, updating the URL to be the URL for your `humanDetection` Function. You can find that URL by visiting the Functions we created above, selecting the correct service, we called ours OutsmartingVoiceMail, and the `humanDetection` Function, clicking the three vertical dots icon, and choosing “Copy URL”. Paste the URL into this widget and click “Save”. !Copy a Functions URL in Twilio - Edit the “pleaseLeaveAVoicemail” widget with your custom message and click “Save”. - Edit the “recordVoicemail” widget so the “Transcription Callback URL” points to either the Voicemail or SMS Function URL, based on how you want to be notified about the voicemail. You can find the URL by visiting the Functions we created above, selecting the correct service, we called ours OutsmartingVoiceMail, and Function, clicking the three vertical dots icon, and choosing “Copy URL”. Click “Save”. - Edit the “thankYouConfirmation” widget with your custom message, click save. 1. Click the red “Publish” button to publish the changes to your flow. 1. Finally, locate the Studio Flow SID for this flow, you can see it in your web browser's address bar, starting with “FW” !Locate a Twilio Studio Flow SID from the URL 1. **IMPORTANT** Revisit the `humanDetection` Function in the service containing the Functions we created above, (I called mine `OutsmartingVoiceMail`), select “humanDetection”, and change the value for the `studioFlowSid` variable in the code to match your unique Studio Flow SID. Click to “Save” your Function, then click “Deploy All”. Keep an eye on the Function logs to make sure the deployment was successful.

Congratulations! You successfully imported your Studio Flow!

What is this Flow doing?

Unlike the Studio flow from the previous blog post on implementing voicemail, this Studio flow will cover the scenario where the dialed end-point has its own voicemail – which they often do. If that device's voicemail kicks in before the Studio “Connect Call To” timeout value, voicemail may be stored on the dialed device's voicemail instead of on Twilio.

We’d prefer that all voicemail be centralized on Twilio so it can be recorded, transcribed, archived and sent via email or SMS.

The Studio flow, “studioLeaveVoicemailBlogWhisper”, addresses this centralized voicemail challenge by requiring the dialed party to positively acknowledge the acceptance of the inbound call when answering, by pressing any key. If the caller doesn’t press a key, we assume it was answered by a machine and divert the call to Twilio’s voicemail.

Note: we are using the Studio TwiML redirect widget (we named redirectDialHumanDetection) along with a Twilio Function to place the call and monitor the DialCallStatus. Twilio Studio does not currently support the TwiML Number URL parameter, which would allow the dialed party the option of positively accepting the call.

“Human Detection” has many possible uses – this is just one example. Another popular use case is urgent alerts, where you want to make sure the dialed party who answers the call is a human before playing a message.

Configure your Twilio number & test it out!

Now that your Studio flow is built, let’s configure your Twilio number to test it out.

Visit the Phone Numbers console first.

If you’re not familiar with how to configure a Twilio number for incoming calls, click on the number you want to use for your “studioLeaveVoicemailBlogWhisper” Studio flow in your Active Numbers here, then scroll down to the “A CALL COMES IN” dropdown in the Voice section and select Studio Flow.

Then, select your new Flow in the Select a Flow dropdown to the right. Follow the same process for “PRIMARY HANDLER FAILS”. Under “CALL STATUS CHANGES”, copy the Webhook URL under your Studio Flow Trigger Widget and paste it into the edit box.

(The last two steps are there to avoid stuck executions.)

Finally, click “Save” at the bottom, and you’ll be all set to test by placing a call to your Twilio number.

Adding a call status change webhook in Twilio

Outsmarting Voice Mail With Human Detection

That’s another wrap! I hope you leave with a firm understanding of optimal call routing using Studio and Twilio Functions to extend the art of the possible.

Human Detection can be put to many uses – many more than the ones I mentioned earlier. Please feel free to further extend what you’ve learned to other use cases.

I can’t wait to see what you build!

Alan Klein is a Principal Solutions Engineer based in Atlanta, GA Office. Alan is a Serverless solution champion and Programmable Voice and Elastic SIP Trunking SIP Expert at Twilio. He's currently focused on furthering his knowledge in Node.js and front-end development and sharing his knowledge with others. You can reach him at aklein [at] twilio.com or on Twitter at @SystemsEng.

Related Resources

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.

Outsmart Voice Mail with Human Detection Using Keypress Prompts

Related Posts

Related Resources

From APIs to SDKs to sample apps

The latest ebooks, industry reports, and webinars

Twilio's developer community hub