Peering Enhances Performance at 415 Million Emails per Hour

Peering Enhances Performance at 415 Million Emails per Hour

This post originally appeared on the Twilio SendGrid blog.

One of the main methods for businesses and consumers to stay connected during COVID-19 has been through email. So when it came to preparing our communication platform for the holiday season, we took this increased responsibility to heart.

For several months before the Black Friday weekend, we tested and tuned our infrastructure. Similar to years past we followed every anomaly, chased down every clue we uncovered to ensure stability, availability, and performance in our platform. We were preparing for holiday traffic that no one could predict with a high degree of certainty. All that preparation paid off—the volume our customers produced over the holiday weekend was unprecedented in the history of Twilio SendGrid.

This year we processed:

• 5.8 billion emails in 24 hours on Black Friday
  • 41% increase over 2019
• 5.7 billion emails in 24 hours on Cyber Monday
  • 36% increase over 2019
• 22 billion emails across the entire Black Friday weekend
  • That’s 2.8 emails per person on earth!
• 415M emails in our peak hour with no measurable increase in our delivery time

The sheer volume of mail delivered means we absolutely smashed the prior record of 116gbps with our new record of 262gbps sustained traffic to inboxes.

Delivery With Peer

This year we made changes at every layer of our stack to ensure the best possible experience for our customers on our busiest weekend. Twilio SendGrid operates its own data centers, which gives us the ability to own and manage every level of our infrastructure. This allows us to tune and optimize the infrastructure to provide best in class performance for delivering email under the highest loads. One of the most significant changes we made this year was at our network layer.

The majority of all email is delivered over the public internet. However, due to our scale and infrastructure, we have begun peering directly with mailbox providers in our network layer. Peering allows two networks to connect and exchange traffic directly without having to carry traffic across the Internet. Private peering provides enhanced reliability and performance advantages for delivery and in-peer networking.

Perhaps the largest advantage of peering is reliability. When we deliver to Google, Yahoo, and Apple inboxes we default to our private fiber circuits, directly from our router to theirs. Not only is this a shorter path on the network, but it’s also a less complex one. Compared to transit over the public internet, there are far fewer routers and links that our bits have to cross to get to an inbox we are peered with. The end result is greater reliability, both against failure and against degraded performance.

Only in the very rare event of both our active and redundant fiber circuits being down do we cut back to transit via the public internet (the default and only way the vast majority of other mail is delivered). This gives us unparalleled reliability for access to the biggest inbox provider, Google, as well as Yahoo and Apple on the Internet Exchange.

Reliability isn’t the only advantage peering provides. After lighting up the fiber to Google, we saw our connection time drop, and significantly at that. Our connection time delivering to Google inboxes is now 21% faster than it was before we lit up our peered circuit. This helped us hit our new Black Friday records.

Peering doesn’t just help with delivery. Our peers with Google and Amazon also ensure greater reliability and performance for our customers operating in their cloud offerings. If you sit inside AWS or GCP and are delivering via roughly half of our points of presence, your mail is arriving to us over one of these private circuits. This means that Twilio SendGrid’s customers who use AWS or GCP receive the same benefits when calling our APIs.

Looking back on 2020

This year has massively changed the way we live, increasing our dependence on technology in our day-to-day lives. Digital communication more than ever dictates how we work, how we forge connections with one another, and how we interact with the companies and the products we consume. In uncertain times like these, reliability and stability become even more valuable as people depend on networked communication for every aspect of their lives.

As I reflect on our 2020 Black Friday and Cyber Monday preparations, I’m proud of our ability to continue to support our customers’ email delivery through unprecedented uncertainty and significant growth.