A benchmark is an external reference value used to compare your performance against other organizations. Boards ask for benchmarks constantly, and the honest answer is usually more complicated than the question expects.
A defined peer set. “Industry average” without a stated population is decoration.
A common definition applied to every contributor. If each organization computed the figure its own way, the average is arithmetic performed on incompatible inputs.
Normalization for size and shape. Absolute counts across organizations of different scale are close to meaningless. See normalized metric.
A stated collection date. Security benchmarks age quickly.
Sample size and how contributors were recruited. Self-selected surveys skew toward organizations willing to report, which is not a random sample of anything.
Most published security benchmarks satisfy two or three of these.
Two banks report vulnerability remediation times. One counts from discovery, one from ticket creation, three days apart. One includes internal systems, one does not. One excludes accepted risks from the denominator, one does not.
The published figures differ by a factor of two and neither organization is performing differently. The variance is definitional.
This is not a flaw in any particular benchmark study. It is structural, and it is why benchmark comparisons should be used to frame a conversation rather than to set a target.
Your own trend is almost always more useful than a peer figure, because the definition is under your control and constant.
Where an external reference genuinely helps, prefer a regulatory or framework expectation over a peer average. It is a defined standard rather than an average of unknown practices.
And where a board wants a peer number, give it with its caveats attached rather than stripped. A benchmark presented without its definitional footnotes will eventually be quoted back at you.
From the blog