Billing Incident Update

July 18, 2013
Written by
Twilio
Twilion

Twilio Bug Logo

At 3:28am PDT/10:28am GMT, our monitoring systems reported an anomaly in our billing systems, which resulted in erroneous credit card charges and in some cases account suspensions. This incident affected 1.1% of customer accounts. The on call team immediately began an incident response, using the Twilio status dashboard at status.twilio.com to update customers at regular intervals. By 6:24am PDT/1:24pm GMT all suspended accounts had had service restored. The same problem briefly re-occurred at approximately 12:30pm PDT/7:30pm GMT, affecting 0.3% of customer accounts, which were immediately remedied.

At this time, the Twilio billing system is offline and account-balances are not being updated in real-time until we fully resolve the issue. We are actively processing credit card refunds, and you will see a transaction void or refund on your credit card statement shortly (most banks process these within 24-48 hours). While the billing issue is being resolved, all voice and messaging services continue to operate normally.

In addition to refunding erroneous credit card transactions, Twilio will also be crediting affected accounts an additional 10% of their last 30 days’ spend. We recognize that it’s not about the money, but our responsiveness to the situation that matters to you. If your account was affected, please consider this credit an acknowledgement of the inconvenience we’ve caused you. If affected customers incurred overdraft or over-limit fees due to this incident, we will also make them whole. We are in the process of contacting all affected customers via email.

Additionally, we will be releasing a full postmortem on the incident once all events and root causes are known, as well as detailing the corrective steps we’ll be taking.

Our focus is on providing you the best quality service and experience, and we recognize that today’s disruption came up short of what you expect from Twilio. Please accept our apologies and know it is our mission to always be earning your trust and business.

As many have you have pointed out on Twitter, this is not a fun day for our team, and especially our engineers. We appreciate your support and your patience.

Sincerely,
Jeff Lawson
CEO & Co-founder

Feel free to email help@twilio.com if you have any questions.