Incident Report: Employee and Customer Account Compromise

Our initial post was published August 7, 2022.

Twilio believes that the security of our customers’ data is of paramount importance, and when an incident occurs that might threaten that security, we communicate what happened in a transparent manner. To that end, we are providing an overview of this incident impacting customer information and our response.

What happened?

On August 4, 2022, Twilio became aware of unauthorized access to information related to a limited number of Twilio customer accounts through a sophisticated social engineering attack designed to steal employee credentials. This broad based attack against our employee base  succeeded in fooling some employees into providing their credentials. The attackers then used the stolen credentials to gain access to some of our internal systems, where they were able to access certain customer data. We continue to notify and are working directly with customers who were affected by this incident. We are still early in our investigation, which is ongoing.

More specifically, current and former employees recently reported receiving text messages purporting to be from our IT department. Typical text bodies suggested that the employee's passwords had expired, or that their schedule had changed, and that they needed to log in to a URL the attacker controls. The URLs used words including "Twilio," "Okta," and "SSO" to try and trick users to click on a link taking them to a landing page that impersonated Twilio’s sign-in page. The text messages originated from U.S. carrier networks. We worked with the U.S. carriers to shut down the actors and worked with the hosting providers serving the malicious URLs to shut those accounts down. Additionally, the threat actors seemed to have sophisticated abilities to match employee names from sources with their phone numbers. 

Sample SMS phishing messages:

We have heard from other companies that they, too, were subject to similar attacks, and have coordinated our response to the threat actors – including collaborating with carriers to stop the malicious messages, as well as their registrars and hosting providers to shut down the malicious URLs. Despite this response, the threat actors have continued to rotate through carriers and hosting providers to resume their attacks.

Based on these factors, we have reason to believe the threat actors are well-organized, sophisticated and methodical in their actions. We have not yet identified the specific threat actors at work here, but have liaised with law enforcement in our efforts. Socially engineered attacks are -- by their very nature -- complex, advanced, and built to challenge even the most advanced defenses.

What have we done?

Once Twilio confirmed the incident, our security team revoked access to the compromised employee accounts to mitigate the attack. A leading forensics firm was engaged to aid our ongoing investigation.

We have reemphasized our security training to ensure employees are on high alert for social engineering attacks, and have issued security advisories on the specific tactics being utilized by malicious actors since they first started to appear several weeks ago. We have also instituted additional mandatory awareness training on social engineering attacks in recent weeks. Separately, we are examining additional technical precautions as the investigation progresses.

As the threat actors were able to access a limited number of accounts’ data, we have been notifying the affected customers on an individual basis with the details. If you are not contacted by Twilio, then it means we have no evidence that your account was impacted by this attack.

Trust is paramount at Twilio, and, we know the security of our systems is an important part of earning and keeping your trust. We sincerely apologize that this happened. While we maintain a well-staffed security team using modern and sophisticated threat detection and deterrence measures, it pains us to have to write this note. We will of course perform an extensive post-mortem on this incident and begin instituting betterments to address the root causes of the compromise immediately. We thank you for your business, and are here to help impacted customers in every way possible.

Next steps

The Twilio Security Incident Response Team will post additional updates here if there are any changes. Also note that Twilio will never ask for your password or ask you to provide two-factor authentication information anywhere other than through the twilio.com portal.

Status Update – August 10, 2022

As a follow-up to our communication regarding the ongoing social-engineering phishing scam that has targeted numerous companies recently, Twilio is continuing its investigation. Security and trust are our top priority as we gather more information. At this time, we can share the following updates:

• We have identified approximately 125 Twilio customers whose data was accessed by malicious actors for a limited period of time, and we have notified all of them
• There is no evidence that customer passwords, authentication tokens, or API keys were accessed without authorization

Our information security team has been working diligently to share details about the attack with impacted customers. If a customer has not been contacted by Twilio, then it means that there is no evidence that their account was impacted by this attack. Our investigation is still ongoing, and if we identify any additional customers that were impacted, our information security team will reach out to them directly. Due to the ongoing and sensitive nature of the investigation, we are not providing further details publicly.

The malicious actors continue to launch social-engineering attacks. However, we have instituted a number of additional measures internally to protect against these attacks, including hardening our security controls at multiple layers.

We are very disappointed and frustrated about this incident. Trust is paramount at Twilio, and we recognize that the security of our systems and network is an important part of earning and keeping our customers' trust. We are committed to learning from this incident and continuing to improve our processes.

We will maintain continuous communication with impacted customers as we move forward with our investigation.

Status Update – August 24, 2022

As we are continuing our investigation and gathering more information, we can share the following update:

After having instituted a number of targeted security enhancements internally, we have not observed any additional instances of unauthorized access to accounts since our last update.

To date, our investigation has identified 163 Twilio customers - out of a total customer base of over 270,000 - whose data was accessed without authorization for a limited period of time, and we have notified all of them.

In addition, to date, our investigation has identified that the malicious actors gained access to the accounts of 93 individual Authy users - out of a total of approximately 75 million users - and registered additional devices to their accounts. We have since identified and removed unauthorized devices from these Authy accounts. Twilio purchased Authy in 2015 and various elements of Twilio’s platform support the functionality of Authy.

We have contacted the 93 Authy users and provided them with additional guidance to protect their account, based on industry-accepted practices:

• Review any linked account(s) for suspicious activity and work with their account provider(s) if they have any concerns.
• Review all devices tied to their Authy account and remove any additional devices they don't recognize.
• To prevent the addition of unauthorized devices, we recommend that users add a backup device and disable “Allow Multi-device” in the Authy application. Users can re-enable “Allow Multi-device” to add new devices at any time. Specific steps can be found here.

Trust is paramount at Twilio, and we recognize that the security of our systems and network is an important part of earning and keeping our customers' trust. As we continue our investigation, we are communicating with impacted customers to share information and assist in their own investigations. We will update this blog with more information as it becomes available.

Investigation Conclusion – October 27, 2022

This is our final update to this blog post describing a security incident involving an SMS phishing (or “smishing”) attack targeting Twilio employees, resulting in unauthorized access to some internal non-production systems. Twilio and a leading forensic firm conducted an extensive investigation into the incident, and we provided updates to our blog as information became available. The investigation has now concluded, and we’d like to share our findings.

Background

In mid-July 2022, malicious actors sent hundreds of smishing text messages to the mobile phones of current and former Twilio employees (the “Smishing Incident”). The malicious actors posed as Twilio IT or other administrators and urged users to click on what appeared to be password-reset and other links. The links led to fake Okta login pages for Twilio. These fake pages were hosted on domains created by the malicious actors, such as twilio-sso.com, twilio.net, twilio.org, sendgrid-okta.org, twilio-okta.net, and twilio-okta.com. Some Twilio employees entered their credentials on these fake pages. The malicious actors then used the credentials of these Twilio employees to access internal Twilio administrative tools and applications to access certain customer information, which we have detailed in previous blog posts on the incident.

Our investigation also led us to conclude that the same malicious actors likely were responsible for a brief security incident that occurred on June 29, 2022. In the June incident, a Twilio employee was socially engineered through voice phishing (or “vishing”) to provide their credentials, and the malicious actor was able to access customer contact information for a limited number of customers. The threat actor’s access was identified and eradicated within 12 hours. Customers whose information was impacted by the June Incident were notified on July 2, 2022.

Smishing incident findings

Our investigation into the Smishing Incident found the following:

• The last observed unauthorized activity in our environment was on August 9, 2022;
• 209 customers – out of a total customer base of over 270,000 – and 93 Authy end users – out of approximately 75 million total users – had accounts that were impacted by the incident; and
• There is no evidence that the malicious actors accessed Twilio customers’ console account credentials, authentication tokens, or API keys.

We have completed our outreach to customers who had affected accounts and worked with them to understand the impact.

Industry findings regarding widespread cyberattacks

Independent cybersecurity researchers examining worldwide cyber incidents have found that a wide-scale set of attacks have been launched against numerous technology companies, telecommunications providers, and cryptocurrency-related individuals and organizations. According to these researchers – who have dubbed the malicious actors “0ktapus” or “Scatter Swine” – the attacks involved (a) identifying the mobile phone numbers of employees at such organizations, (b) sending smishing texts or making voice phishing (“vishing”) phone calls to identified phone numbers to trick the employees into clicking on links that led to fake Okta, Azure, Duo and other login pages, (c) harvesting the employees’ credentials and one-time passwords (“OTPs”) through those fake pages, and (d) using those credentials to advance reconnaissance operations within the target networks in order to attempt user account takeovers and further smishing efforts targeting other organizations. The attack on Twilio employed similar tactics, techniques, and procedures.

Eradication and remediation efforts

Upon discovering the unauthorized access to our systems, Twilio took a number of actions to eradicate the malicious actor’s access during the Smishing Incident, including:

• Resetting credentials of the compromised Twilio employee user accounts;
• Revoking all active sessions associated with the compromise of Okta-integrated apps;
• Blocking all indicators of compromise associated with the attack; and
• Initiating takedown requests of the fake Twilio domains.

To prevent or mitigate the efficacy of similar smishing and vishing attacks in the future, Twilio has also implemented a number of additional security measures, including:

• Implementing stronger two factor precautions and distributing FIDO2 tokens to all employees;
• Implementing additional layers of control within our VPN;
• Removing and limiting certain functionality within specific administrative tooling;
• Increasing the refresh frequency of tokens for Okta-integrated applications;
• Conducting supplemental mandatory security training for all employees regarding attacks based on social engineering techniques.

Conclusion

We’d like to apologize to our customers for the incidents. We have talked to hundreds of customers, conveyed our regrets, and described our ongoing efforts to improve. We deeply appreciate the understanding and support that customers have shown, and we’ve shared our commitment to do better. We are seeing immediate benefits from the significant enhancements we have made to our security posture, and are making long term investments to continue to earn back the trust of our customers.