Security Metrics that Count

Security Metrics that Count

Quantifying security has been a hot topic for a long time, and it is extremely difficult to do. Unlike other areas of a technology-based company, security isn’t typically a revenue generator, so it is often only viewed as a cost center. A security-based organization prevents bad things from happening to their customers and their customers' data, but it's hard to quantify things that don’t happen. We on Twilio’s Security team have been working to figure out the best metrics to successfully portray our security capabilities, and show how the changing security posture of the company is contributing to our revenue.

Our vision is to use security metrics to drive change within the organization, to celebrate improvements over time which help us better protect Twilio’s customers, and to measure our security program. We want our security metrics to both help Engineering teams and help us understand the maturity of our security program. In this post, we’ll share our insights into what security capabilities we believe provide more value than others, and what we’ve found is worth the extra manual effort.

Who is the audience for security metrics?

While we conducted our metrics research, we discovered that different audience groups within Twilio were interested in different kinds of security metrics. Executive-level leadership wanted to understand the security posture across the organization, VPs wanted to understand the security posture of their specific business units, product managers wanted to understand the security posture of their products, and engineering managers wanted to understand how many open vulnerabilities were present and which ones their teams should prioritize fixing.

In our research, we found that many teams and contributors were excited to receive a report of our security metrics – and we bet your company will have a similar experience.

Which security metrics matter the most?

We set out to tell two different stories about the security health of our organization and the maturity of the security program. These two stories require very different metrics.

In the table below, we tried to capture all of the different metrics that we use or plan to use:

Setting a baseline

To show the security posture of our products, we needed information about vulnerabilities in our products. To show the maturity of Twilo’s security program, we needed information about the different security activities performed in the product life cycle. Luckily for our team, both of these pieces of information were present in Jira, our issue management system.

When we first started our evaluation, we looked at our existing vulnerability tickets and identified additional pieces of information that would be helpful to generate metrics. From there, we worked with our IT team to add these metrics to the vulnerability ticket template, which included information about the business units the vulnerabilities fell under, as well as the vulnerability source and category.

The architecture for high-level metrics generation

Getting the security data to generate metrics, keeping those metrics up to date, and customizing them for different audiences meant that we had to automate this process end-to-end.

We started by identifying our source of truth (data in Jira) to get the vulnerability data, and built a cron job using Python to normalize the data. We built automation using Python which periodically  collected data from Jira and transformed it to specific fields needed for metrics. This normalized data was then fed into a Google sheet which served as the data source for Google Data Studio, where we were able to customize dashboards for different audiences.

Interpreting security metrics

How do you show progress over time? We set out to look not only at point-in-time metrics of how many Critical/High issues were open, but also to look at a trend of vulnerabilities open over time. Ideally, we wanted to see that we were working towards closing vulnerabilities faster than we were finding them.

The key indicator in our effort is progress over time and not a blanket “Close all vulnerabilities” approach, since  eliminating all risks is not practically possible and we want to strive towards reducing risk to the company over time to a manageable level.

Tailoring security metrics to different audiences

Our metrics had to provide a 10,000 foot view for presentations to leadership, and also offer a breakdown of data for anyone who wanted to dig into the details.  

To address this requirement, we created multiple pages within the same dashboard in Google Data Studio. Each page contained information specific to an audience.

The screenshot shown below is of a dashboard that shows the security posture of individual BUs (Business Units), with filters for the severity of vulnerabilities:

The screnshot below is a birds'-eye-view of the company’s security posture and shows how different business units contribute to the open vulnerabilities.

Using metrics to shift left

What security blog post would be complete without a few buzzwords? 

The metrics that tracked the maturity of the security program provided the Product Security team insights into how different capabilities such as SAST, DAST, penetration tests, threat modeling, bug bounty program added value to the SDLC process, and at which stage of the SDLC the majority of the vulnerabilities were being discovered.

Using this information, the Product Security team started working towards shifting their vulnerability discovery approach so that the bulk of the vulnerabilities found are in the design and coding phases of the SDLC (software development lifecycle).

What we achieved by reimagining our security metrics

Reimagining our security metrics was a great success. During the project, we found: :

• Metrics and visibility help drive change. We saw teams actively take ownership of open vulnerabilities and fix them once they had seen our reports
• Gamifying team activity to respond to the insights unveiled by our new security metrics helped create a healthy competition between Twilio teams, with teams competing to see who could fix the most vulnerabilities
• Additions to the metrics dashboard could cause some issues. Whenever we added a tool into the metrics, the dashboard would show a spike in the metrics. It was important to point out that the spike was not a downward trend in our security tracking, merely an added layer of visibility
• Security metrics should help identify trends of vulnerabilities present over a period of time. In other words, are we fixing vulnerabilities faster than they are generated?

View Twilio’s security metrics template in Google Data Studio

We have a public Google Sheet with sample data, and the corresponding Google Data Studio dashboard. We would love for you to take a look and play around with our templates (please make a copy before modifying them) and do let us know what you think.

Our team found this exercise incredibly valuable, and our partners at Twilio were thrilled with the new visibility. We can’t wait to see how your company finds the approach!

About the Authors

Harini Rangarajan manages the Product Security Team at Twilio, helping build secure products at scale. She can be reached at hrangarajan@twilio.com.

Yashvier Kosaraju is the interim Director of Product and Infrastructure Security at Twilio, where he helps run globally distributed teams. He can be reached at ykosaraju@twilio.com.