A metric is reproducible when someone can recompute it later and get the same answer, without access to the person who produced it originally. It is the property that separates a measurement record from a collection of past assertions.
The source records for the period, retained in the form they arrived.
The formula, versioned, with the version that produced each value recorded against it.
The parameters in effect at the time: thresholds, scope, exclusions, timezone.
A computation process that does not depend on undocumented steps living in one person’s working method.
Miss any one and reproducibility fails. Most programmes miss the first and the fourth.
Choose a metric from a board pack eighteen months old. Ask someone who did not produce it to reproduce the figure from what the organization has retained.
The outcome is one of three. They reproduce it, in which case the programme is in good shape. They produce something close but not identical, which means undocumented judgement was involved. Or they cannot start, because the source extract was overwritten and the spreadsheet has been edited nineteen times since.
The third is the common result, and it is usually discovered during an audit rather than during a drill.
Reproducibility is invisible until it is urgently needed, which is why programmes go years without it.
The moments it gets tested are predictable. A regulatory examination covering prior periods. An audit challenging a reported figure. A board member noticing a trend break. A dispute with a business unit over their number. Due diligence during a transaction.
In every one of those, the inability to reproduce is read as a control weakness rather than as an inconvenience, and it contaminates confidence in metrics that were perfectly sound.
Spreadsheets fail all four requirements simultaneously, which is why programmes that look healthy at the reporting layer turn out to be hollow underneath.
The file gets copied and edited. The extract it depended on is gone. The formula exists in several versions across several copies with no way to tell which produced the reported number. And the process lives in the head of whoever built it.
None of this is visible while the person is still there. See shadow spreadsheet.
From the blog