Microservice for Handling SendGrid Event Webhooks

The first part of this blog series showed how enterprises could build a microservice to send outbound emails in a centralized and scalable way.

In this second post on AWS microservices, we will handle incoming SendGrid Event Webhooks. These Event Webhooks are feedback from the engagements generated by outbound emails sent via the SendGrid Email API.

This post – and yes, all three blog posts – use SAM (Serverless Application Model) templates. Basic familiarity with AWS is all that is required to spin these up. GCP and Azure users should benefit from these templates too, though, as they should chart an effective course for building similar functionality in those platforms.

Be sure to check out the entire three part SendGrid Microservices Blog Series:

1. Serverless Microservice for Sending Emails using SendGrid Email API
2. Serverless Microservice for handling SendGrid Event Webhooks
3. Serverless Microservice for handling SendGrid Inbound Parse

There are three distinct parts to this solution: the blog, the companion video, and heavily-commented code. Each part offers something different so be sure to review all three.

Part 2: Serverless Microservice for handling SendGrid Event Webhooks

Let’s get started!

Learning by video can be effective. You can watch the video before proceeding or come back to it later!

Here is what we are going to spin up in part two of the series:

Let’s walk through the flow starting from the top left.

A. SendGrid Sends Emails and Events are Generated

After SendGrid receives an API request to send an email, it quickly processes the request payload and delivers the email(s) to the Inbox Providers (GMail, Outlook, Yahoo, etc.) who, in turn, deliver them to the email End Users.

Through the journey of those emails, events are generated by the Inbox Providers (delivered, bounced, etc.) and the email End User (open, click, etc.) and sent back to SendGrid.

As a SendGrid customer, you are able to consume these events and have the data sent back to your systems for processing via Webhooks.

B. Receive the Event Batch from SendGrid

Event Webhooks can have spiky demand. You need scalable systems in place to be able to handle incoming requests and place them in a queue for additional processing.

This template offers two approaches to handling incoming events from SendGrid.

1. Events go from API Gateway directly into an SQS Queue.
2. Events go from API Gateway to a Lambda function which dumps the requests into an S3 bucket. The S3 bucket triggers an event which feeds into the SQS Queue.

Which approach should you use? First, you should know that SendGrid batches events as it receives them and sends them to your endpoint at regular intervals or when batches get too large.

Approach A is the simplest, but because SQS has a maximum payload size, this option could not be effective for heavy / spiky volume.

Approach B is likely best for most cases. The batches from SendGrid are dumped directly into an S3 bucket so there is no concern about maximum payload size. The SQS queue is still utilized to control flow to downstream processors.

C. Process Event Batch

A Lambda function is attached to the SQS Queue. You have the ability to control the rate in which messages are consumed from the queue.

The first order of business for this Lambda is to validate that each request actually came from SendGrid. This is done by taking request headers x-twilio-email-event-webhook-signature and x-twilio-email-event-webhook-timestamp and validating them along with the key set in your SendGrid Console. All validated requests proceed while unvalidated requests are blocked and can trigger alarms or additional processing.

The payload is then pulled from the SQS message or from S3 (depending on the A or B choice in section B above), and then the array of events from SendGrid are separated out and each event is published separately to the SNS topic for additional processing by downstream consumers.

D. Additional Processing

This template has 3 separate “processing” AWS stacks that you can spin up to handle events published to the SNS topic:

1. A Lambda function that saves each event to an S3 bucket.
2. A Lambda function that saves each event to DynamoDB.
3. A Lambda function that is a simple stub for you to handle however you need!

If you have read this far, then I think you are ready to see how you can spin this up yourself!

Let’s get started…

AWS Resources

All of the AWS components of the serverless application are provided as "Infrastructure as Code" (oft-shortened to IaC) and deployed via CloudFormation into AWS. Here is an overview of the components:

• AWS SAM => an open-source framework that enables you to build serverless applications on AWS
• AWS CLI => a command line interface to AWS services. Not required, but recommended because it simplifies credentials when deploying
• AWS Lambda => serverless compute service
• S3 => Object Storage
• SQS => Simple Queue Service
• SNS => Simple Notification Service

Prerequisites

This is not a beginner level build! You should have some knowledge of AWS, serverless computing, and programming.

If you have already completed part 1 of this series, then you should already have these in place!

• Twilio SendGrid Account. If you don’t yet have one, you can sign up for a free account here.
• An authenticated domain and a verified sender in your SendGrid account.
• AWS Account with permissions to provision Lambdas, step functions, S3 buckets, IAM Roles & Policies, an SQS queue, and SNS topics. You can sign up for an account here.
• AWS CLI installed with AWS credentials configured.
• AWS SAM CLI installed.
• Node.js installed on your computer.

Let’s Build it!

Here are the basic steps of our build today.

1. Set up an Event Webhook from your SendGrid Account
2. Download the code
3. Deploy the stack
4. Update the Webhook back in SendGrid
5. Try it out!

1. Set up an Event Webhook from your SendGrid Account

From your SendGrid Console, go to SETTING => MAIL SETTINGS => EVENT WEBHOOKS.

Click on the button that says CREATE NEW WEBHOOK.  The screen should look like this:

Enter a Friendly Name of your choosing and then put in a fake Post URL (we will generate the real Post URL shortly) and then return here to update the Event Webhook.

You can select which events you wish to receive. For simplicity I am just going to receive Opened and Delivered events but there are many events that you are able to track. Note that paid Email API plans can have multiple Event Webhooks!

Next, scroll down the page and enable SIGNATURE VERIFICATION. Save the Event Webhook and then click on EDIT to return to this view and note the new VERIFICATION KEY. Copy this as you will need to enter it shortly.

2. Download the Code for this Application

Download the code from this repo and then open up the folder in your preferred development environment.

Note that this repo has an aws_stacks folder which contains 4 separate microservices. SendGridEventWebhookHandler is the primary stack and must be installed first.

The other 3 stacks (SaveEventWebhookToS3, SaveEventWebhookToDynamo, and GenericSendGridEventHandler) consume events that come in. You can install one of them or all of them.

Open up a terminal window and go into the primary stack aws_stacks/SendGridEventWebhookHandler.

First we need to install a SendGrid node package to help us validate incoming requests. From that parent directory, cd into these two directories and install the packages. Here are the commands:

The command sequence should look like this:

Now we are ready to enter our SendGrid credentials…

3. Enter your SendGrid Account Credentials

Open up the file template.yaml in the parent “SendGridEventWebhookHandler” directory. This yaml file contains the instructions needed to provision the AWS resources.

As mentioned in step 1 above, we are going to use the key set in the Twilio Console to validate all requests coming into this endpoint. We can also use Basic Authentication. In the template.yaml file, use FIND and search for EVENT_WEBHOOK_USER. Set your own user / password combination and note that for anything other than your initial testing, be sure to save credentials using more secure means like AWS Parameter Store or Secrets Manager!

We will add in this username and password combination along with the endpoint we generate in a subsequent step.

Next, use FIND to search for SENDGRID_WEBHOOK_PUBLIC to bring up this block of yaml code:

This code enables you to set up both types of endpoints. For the initial test use the first one (SENDGRID_WEBHOOK_PUBLIC_KEY_API_TO_S3) and enter the key that you created in step 1 above when you set up the Event Webhook in the SendGrid Console.

4. Deploy Code

With those settings in place, we are ready to deploy! From a terminal window, go into the parent (aws_stacks/SendGridEventWebhookHandler) directory and run:

This command goes through the yaml file template.yaml and prepares the stack to be deployed.

In order to deploy the SAM application, you need to be sure that you have the proper AWS credentials configured. Having the AWS CLI also installed makes it easier, but here are some instructions.

Once you have authenticated into your AWS account, you can run:

This will start an interactive command prompt session to set basic configurations and then deploy all of your resources via a stack in CloudFormation. Here are the answers to enter after running that command (except, substitute your AWS Region of choice):

After answering the last questions, SAM will create a changeset that lists all of the resources that will be deployed. Answer “y” to the last question to have AWS actually start to create the resources.

The SAM command prompt will let you know when it has finished deploying all of the resources. You can then go to your AWS Console and CloudFormation and browse through the new stack you just created. All of the Lambdas, Lambda Layers, S3 buckets, IAM Roles, SQS queues, SNS topics are all created automatically. (IaC – Infrastructure as Code – is awesome!)

Also note that the first time you run the deploy command, SAM will create a samconfig.toml file to save your answers for subsequent deployments. After you deploy the first time, you can drop the --guided parameter of sam deploy for future deployments.

5. Update the Webhook back in SendGrid

With the primary stack deployed, we can now update the SendGrid webhook with the newly created endpoint.

In the AWS Console, go to CloudFormation and then select the Stack you just created (SendGridEventWebhookHandler). Click on the Outputs tab to reveal key::value pairs generated by this stack.

We are looking for these two key::value pairs:

• SGEventWebhookToS3Api
• SGEventWebhookToSQSApi

These two options correspond to the two types of endpoints as described in top introduction section B earlier in this blog post. To recap, one endpoint takes requests and dumps them into an S3 bucket while the other dumps the requests into an SQS queue. You can review top introduction section B above for more details but you can use either or even both options.

For this blog, we will proceed with SGEventWebhookToS3Api. The value for that key should look something like this:

To add in Basic Authentication, we add the username and password combination entered in the template.yaml file in section 3 above to the beginning of the url. You would use your own username and password but the resulting url should look like this:

These endpoints require specific paths added to the base url (SGEventWebhookToS3Api adds /sendgrid-events and SGEventWebhookToSQSApi adds /twilio-sendgrid). These paths are set in two different places. The path for SGEventWebhookToS3Api is set on the “Event” of the lambda function in the template.yaml file. The path for SGEventWebhookToSQSApi is set in the api.yaml file.

So our final URL using using the SGEventWebhookToS3Api key would be:

Now we return to the SendGrid Console and go to SETTING => MAIL SETTINGS => EVENT WEBHOOKS. Select the edit option for your Event Webhook and then paste your new endpoint into the Post URL field as shown below:

Save your changes and you are now ready to receive email events from SendGrid!

6. But what do we do with the events?

We can receive all of these events now, but we have not set up any processors.

As discussed in the introduction section D above, this repo comes with three processors. You can choose which ones you want to spin up, but these instructions will show you how to quickly install all three.

Open a terminal window and go into the aws_stacks/ folder and then enter the following:

6. Try it out…

If you started with the first blog post in this series, you can use that to send an email. If not, send an email via your SendGrid account and you should start receiving events immediately and be able to with them in your processors.

From the sample Email API payload below, this is what you should expect to see after sending an email via your SendGrid Email API:

Processor: Save to S3 Bucket

In your AWS Console, go to S3 and then go into the S3 Bucket that starts with “twilio-sendgrid-event-webhook…”. You should see a folder with a date in yyyy-mm-dd format. Open that folder and you will see additional subfolders for each event type.

You can go into any of those directories and inspect the JSON files for each event. They will look like this:

Note that “categories” and the custom arguments (customArgs) submitted with the request are carried through to the Event Webhook along with other expected key/value pairs so you have everything you need to tie this event back into your profile for this customer and into your analytics and data warehouse infrastructure.

Processor: Save to S3 DynamoDB

From your AWS Console, go to DynamoDB and then select “Tables” and then choose the table that starts with “SendGridEventWebhookDynamoDBTable…”. Then click on Explore Items or click the button that says Explore Table Items.

You will then be able to browse through all of the events that have been added to this table. The template sets the primary key as the “to” email address and the sort key is a concatenation of the event description, a timestamp and the SendGrid event id. You can of course change this to be whatever you want, but this configuration will allow you to make some initial queries.

An item will contain all of the other parameters included in the event:

Processor: Generic Lambda Handler

The last processor is just a simple Lambda function that just writes the inbound event to logs. This is a stub for you to build whatever functionality you need:

• Update your CDP, CRM, or other data sources.
• Handle a bounce or spam report.
• Handle an error.
• Update a dashboard.
• Trigger other events.

Open up Lambda in your AWS Console and then click on the function that starts with GenericSendGridEventHandl-SendGridGenericHandlerFu….

You can look at the code of the function and see that it is pretty simple as it just sends the incoming event to console.log. Click on the MONITOR tab and then click VIEW CLOUDWATCH LOGS. This will take you to CloudWatch and let you see the output from the Lambda function.

Again, this function is just a stub so the logs in CloudWatch will just show the console.log’s as shown below. It is up to you to handle these events however you need to!

Cleanup

To avoid any undesired costs, you can delete the application that you created using the AWS CLI and the console.  

First, delete the S3 buckets. From the S3 “home” page, first empty the buckets by selecting the bucket and then select the EMPTY option. Next, delete the actual bucket by selecting the bucket and then the DELETE button.

Next, delete the stack from CloudFormation in the AWS console. Select the DELETE STACK option. AWS SAM also creates a stack and an S3 bucket to manage deployments. You can delete them from CloudFormation in the AWS console following the same procedure as above.

Deploy to production

While you can get this system working pretty quickly, it is not ready for your production environment. Be sure to customize and refine this codebase for your needs, thoroughly test your version of this system in your environment, and configure tests, error handling, and anything else required by your development standards.

Conclusion

In short order, you now have the foundation for a serverless microservice for handling email events from Twilio SendGrid Event Webhooks!

The code in the repo is heavily-commented. Dive in and see how you can configure your enterprise systems to leverage this event-based microservice AND how you can use SendGrid’s robust tracking capabilities to get a better sense of what you are sending and how your users are engaging with your emails.

Speaking of engaging with your emails, be sure to check out part 1 and part 3 of this blog series to learn about sending emails with SendGrid, as well as Inbound Parse!

***

Dan Bartlett has been building web applications since the first dotcom wave. The core principles from those days remain the same but these days you can build cooler things faster. He can be reached at dbartlett [at] twilio.com.