Late-arriving data is any record that reaches the measurement system after the period it belongs to has already been computed. It is not an edge case. In a security estate it is the normal condition.
An offline laptop reports its state when it reconnects a week later.
A scanner completes a long-running job and publishes results dated to the start of the scan.
An integration fails over a weekend and replays on Monday.
A ticket is closed retroactively with the actual resolution date rather than the entry date.
An acquired subsidiary’s data is onboarded three months after the period it covers.
Any number computed at period close is provisional. A coverage figure produced on the first of the month will move as devices that were offline check in.
Two failure patterns follow. Either the number is frozen at close, in which case it is permanently wrong. Or it moves silently, in which case the figure someone presented last week no longer matches the system, and nobody can say why.
The usual answer is to declare a settling period during which a bucket remains open for revision, and after which it is sealed.
Its length should come from observed behaviour rather than convenience. Look at how long it actually takes for 95 percent of records for a period to arrive, per source, and set the window from that. Sources will differ by an order of magnitude, and the composed metric settles at the pace of its slowest input.
Values inside the window get marked provisional. Values past it are stable and any change to them is a restatement rather than a settlement.
The event time, meaning when the thing happened, and the ingestion time, meaning when the system learned about it. Both, always.
With only one of them, you cannot distinguish a genuine change in the world from data catching up, and those demand different responses. With both, lateness itself becomes measurable, and a source whose lag is growing becomes visible before it distorts anything.
From the blog