A phishing metric trended reassuringly downward for six months. Then someone pulled the query and discovered it had been silently excluding an entire mail gateway. The number never broke. That was the problem.
For six months, a very large enterprise’s security team published a phishing click-through rate that trended reassuringly downward. Executives cited it in board decks. The awareness program built its quarterly narrative around it. Then a new analyst pulled the underlying query and discovered the metric had been silently excluding an entire mail gateway since a source migration eighteen months earlier. The formula’s original author had left the company in early 2024. Nobody who remained could explain why the exclusion existed, whether it had ever been correct, or when it had quietly become wrong. The number had been reporting the whole time. It just wasn’t measuring what anyone thought it was.
This is the failure mode most security programs never see coming, because the metric never breaks loudly. It keeps producing a value. Dashboards keep rendering. The pipeline keeps running. What decays is the semantic contract between the formula and the reality it was supposed to describe, and that decay accelerates every time a data source is renamed, a log schema shifts, a tool is swapped out, or an integration is rewired. Each change is individually defensible. Cumulatively, they turn a once-precise measurement into a plausible-looking artifact that nobody can trace back to a decision.
The root cause is almost never technical. It’s the absence of two artifacts that audit-grade measurement demands and most programs skip entirely.
The first is a named owner on every KPI — a specific human, not a team inbox, who is accountable for the definition and required to review it on a documented cadence. The second is a versioned history of the formula itself, including who approved each change, what business question it was meant to answer, and what data sources it depends on. Without those, a KPI is not a metric. It’s a rumor with a chart attached, and rumors do not survive board scrutiny.
Consider the pattern we see repeatedly in maturity assessments. A CISO commissions a metric in year one to answer a specific question — say, mean time to remediate critical vulnerabilities on internet-facing assets. The engineer who builds it understands the nuance: which asset inventory is authoritative, which severity taxonomy is being used, how remediation is confirmed. Two years and one platform migration later, the asset inventory has been replaced, the severity mapping was quietly updated, and the confirmation logic was rewritten to accommodate a new ticketing system. The metric still reports a number. The number no longer answers the original question. And because no versioned definition exists, no one can reconstruct what the answer used to mean or when it stopped meaning that.
The consequence isn’t just a bad chart. It’s a governance failure. When the board asks why a trend is moving, three stakeholders give three different explanations because each is reasoning from a mental model of the metric that no longer matches its implementation. Auditors flag it. Regulators, increasingly, ask for lineage documentation that programs cannot produce. The security team’s credibility on measurement — arguably the most important currency it has with the business — erodes not through any single mistake, but through the slow drift no one was assigned to prevent.
A named owner reviews the definition on a schedule, signs off on every change, and is on the hook when a source migration would break the semantic contract. A versioned definition means that when a metric moves, you can prove what it measured yesterday, what it measures today, and who authorized the difference. This is the difference between a security program that survives turnover and one whose numbers expire quietly the moment the person who understood them walks out the door.
At Metric Maestro, we build the ownership, versioning, and lineage layer that turns security KPIs into defensible measurements. If your dashboards would struggle to answer who signed off on the formula and when, we should talk before the next board meeting does the asking for you.
Whitepapers
In-Depth Comparisons
Metric Maestro vs Archer GRC
Archer is built for enterprise risk management. Metric Maestro is built for security leaders who need to prove the value of their program to the board.
Metric Maestro vs DIY Security Reporting
Most security teams start with spreadsheets. At some point, the cost of that choice becomes impossible to ignore.