Cybersecurity Performance Metrics: A Work in Progress
 

How well are today’s cybersecurity defenses performing? That’s a hard question to answer — particularly since the security profession lacks consensus on performance metrics.

Yes, there are cybersecurity performance metrics for gauging the effectiveness of specific tools, systems, and approaches, but no common language to compare and contrast among them. Which truly works — and how well?

The problem is that in some cases, industry groups don’t agree on a basic taxonomy of terms. Another factor is that various vendors use disparate measures. It all makes interpreting data extraordinarily difficult, with the range of stakeholders drawing different conclusions.

Closing the Cybersecurity Performance Metrics Gap

Public and private sector organizations are attempting to tackle parts of this challenge. The National Institute of Standards and Technology (NIST) gathered public comment in 2020 on a yet-to-be-released version 2 of its “Performance Measurement Guide for Information Security,”[1] and NIST is including measurement among the topics in the NIST Cybersecurity Framework 2.0, now being drafted with private sector input.[2] Another example is the International Standards Organization’s key performance indicators (KPIs).[3] 

In the private sector, the Gartner research and consulting firm, which has already socialized a general framework of performance metrics (described below), has expanded on that initiative by inviting broad participation in an effort to develop what it calls “cybersecurity value delivery benchmarks.”[4] And OpenVPN, an open source project, presented proposed metrics at an industry conference this year.[5]

But NIST itself has asserted that, “Even as cybersecurity-based risks and costs are increasing, measuring cybersecurity remains an under-developed topic – one in which there is not even a standard taxonomy for terms such as ‘measurements’ and ‘metrics.’”[6]

Finding a solution won’t be easy. Meanwhile, companies are left to navigate their own path.

What’s at Stake for Security Professionals

Lacking standard metrics, “Most organizations have terrible metrics that even they don’t think have much value,” said Gartner Vice President Paul Proctor.[7] The Accenture management consultancy noted that, “It is likely system admins, product owners, application developers, and business leaders will all have a different understanding of certain terms.”[8]

As a result, it’s difficult to gain insight into the actual performance of a cybersecurity system — from the sum of its parts to the whole of the system. This includes routine technical measurements and key performance indicators (KPIs) that rate how well companies are identifying, protecting, detecting, responding, and recovering from cybersecurity attacks.

In contrast, Gartner said, “Imagine a future where you can do peer comparisons for cybersecurity outcomes like time to close incidents, speed to patch systems, third-party risk, endpoint protection, phishing scores, cloud security configuration drift, and zero trust implementation. Combining this data with maturity and spending benchmarks gives you absolute control over your cybersecurity investments and reporting for all your key stakeholders.”

Framing Cybersecurity Performance Metrics

Gartner has proposed a four-part cybersecurity performance measurement framework, based on the acronym CARE.[9]

• Consistency metrics: The goal for these metrics is to determine whether security controls are working consistently over time across the organization. These include third-party risk assessment and security awareness, such as training for spotting phishing messages.
• Adequacy metrics: These metrics assess whether controls meet business needs and stakeholder expectations. They revolve around percentages for factors such as how well an organization addresses patching and how consistently it applies anti-malware defenses to endpoints.
• Reasonableness metrics: The purpose of these metrics is to validate that security controls are appropriate, fair and moderate, measured by their business impact. This includes effects such as delays, downtime, and user complaints.
• Effectiveness metrics: These metrics are designed to assess whether security controls are leading to desired outcomes. They include vulnerability remediations and the prevalence of cloud security incidents. 

Gartner’s framework provides a general template, and its benchmarking initiative aims to help fill out the framework.

There is also an element of performance evaluation incorporated into risk management frameworks, such as the Cybermaturity Platform from ISACA, an IT professional organization.[10] 

In a more general sense, Accenture advocates performance reporting structures based on both a top-down and bottom-up approach. The former revolves around a company’s overall strategy, while the latter involves a co-journey between business and security practitioners with a focus on operational and risk perspectives. The key, Accenture noted, is creating consistent communication across an organization.

Stopgap Measures for Security Teams

Short of widely accepted and standardized cybersecurity metrics and KPIs, which aren’t likely to appear anytime soon, organizations should try to develop their own consistent framework. Goals should include reducing subjectivity — starting by identifying cybersecurity performance metrics that are easy and inexpensive to gather; can be expressed as a single unit of measure, such as a percentage or other numerical figure; and are contextually significant, meaning they can be directly applied to the decision or action necessary at that moment.

Integrating an organization’s various point security solutions and endpoints onto a single platform empowered with data analytics is key to gathering insights and analyzing performance. To this end, security vendors such as Mimecast are rapidly implementing integrated platforms that also support extended detection and response (XDR), which collects and organizes critical data across email, endpoints, security tools, the network and the cloud.

The Bottom Line

Standard cybersecurity performance metrics remain a work in progress — an important work to help guide decision-making and improvements in battling the ongoing wave of cyberattacks. As leading organizations try to build consensus, companies should address their own internal inconsistencies for better cybersecurity outcomes and more effective communication across their organizations. See how Mimecast’s integrated data analytics can assist in the measurement process.

[1] “Pre-Draft Call for Comments: Performance Measurement Guide for Information Security,” NIST

[2] “Journey to the NIST Cybersecurity Framework 2.0,” NIST

[3] “Measuring ISO 27001 ISMS Processes: A 5 Step Guide,” ISO27001 Guide

[4] “Benchmarking Cybersecurity Value Delivery,” Gartner

[5] “Why Metrics Are Crucial to Proving Cybersecurity Programs’ Value,” CSO Online

[6] “Cybersecurity Measurement,” NIST

[7] “Benchmarking Cybersecurity Value Delivery,” Gartner

[8] “How to Effectively Use Metrics for Security Optimization,” Accenture

[9] “4 Metrics That Prove Your Cybersecurity Program Works,” Gartner

[10] “Cyber Risk Assessment is Just the Beginning,” ISACA.