Auditable or Best-Effort: The Test Every Security Metric Must Pass

Every security number that leaves your team lands in one of two buckets, and most organizations cannot tell you which bucket a given KPI belongs to until someone external forces the question. That is the tension we want to name plainly. A metric is either auditable — produced by a defined formula, run on versioned data, with a timestamp and a named owner — or it is best-effort, which is a polite way of saying whoever had time on Thursday pulled something together in a spreadsheet nobody has opened since. Both numbers can look identical on a slide. Only one of them survives a follow-up question.

Why the Regulatory Clock Is Running

The distinction used to be a matter of internal hygiene. It is not anymore. Regulators reviewing SEC cyber disclosures, DORA operational resilience filings, and NIS2 reporting, as well as the tightening ISO 27001:2022 measurement clauses, are all asking the same thing in different accents: show us how you derived this number, on what data, at what point in time, under whose sign-off. Auditors trained on financial controls are applying the same reproducibility standard to security telemetry, because the disclosure risk is now equivalent. Boards have followed suit. A number that cannot defend its own provenance is treated, correctly, as a claim rather than evidence.

The Ninety-Four Percent Problem

Consider a common example. A CISO reports that ninety-four percent of critical vulnerabilities were remediated within SLA last quarter. Auditable means we can point to the vulnerability management system of record, the exact query that filtered on severity and asset scope, the snapshot of the ticket data used, the SLA definition in force during that window, and the reviewer who approved the calculation. Best-effort means an analyst exported a CSV, deleted a few rows that looked like duplicates, applied a filter they cannot precisely reconstruct, and rounded. Same headline, radically different exposure. When a regulator or a litigator asks how the figure was produced, only one version has an answer that does not require an apology.

Where the Gap Keeps Widening

The gap widens across the metrics stack. Mean time to detect, phishing simulation failure rates, third-party risk coverage, control effectiveness scores, patch cadence, identity hygiene — every one of these can be sourced from live systems with a versioned formula, or approximated by someone under deadline pressure. The industry has quietly tolerated the second mode for years because the first mode was genuinely hard to build. Data lived in twenty tools, ownership was unclear, and the definitions drifted between quarters. That tolerance is expiring, and the organizations still relying on manual assembly are discovering that “we generally track this” is not a defensible position in front of a regulator or a plaintiff’s expert witness.

Fewer Numbers, Each Backed by a Pipeline

Our position at Metric Maestro is straightforward. If you cannot point to the source, the formula, the freshness, and the owner behind a KPI, you are shipping best-effort, and you should say so internally before someone else says it externally. That is fine on a Tuesday when the number is informing an internal decision. It is not fine on a Friday when the board, the auditor, or the SEC asks how you arrived at it. The remediation is not more reporting. It is fewer numbers, each backed by a defined pipeline, a versioned definition, and a person whose name is attached to the calculation.

We built Metric Maestro to make that upgrade practical rather than aspirational — to turn security metrics into evidence that holds up under a board question, an audit, and the next regulatory letter. If you are not sure which of your KPIs would survive that test today, we would like to help you find out before someone else does. Talk to us.