Time is an opinion until you discipline it

Every machine's clock is wrong. A decent quartz oscillator drifts on the order of parts per million, which sounds negligible until you turn it into a day: a few parts per million is a handful of milliseconds every hour, accumulating quietly in one direction. Left alone for a week, two machines that agreed on Monday are describing different worlds by Friday. This is not a fault condition anyone gets paged for - it is the default behavior of computers, and it is the reason clock synchronization is a correctness requirement in a distributed system rather than a housekeeping task.

The property at stake is ordering. If a node in one region stamps an event at 14:02:11.100 and a node in another stamps a related event at 14:02:11.040, the sequence you read from those numbers is only real if both clocks are disciplined against the same reference to a tolerance smaller than the gap. Otherwise you have not observed a sequence, you have observed two opinions. NTP over a well-connected path holds machines within single-digit milliseconds; PTP with hardware timestamping does far better. What matters is not which protocol but that the tolerance is known, monitored, and smaller than the intervals anyone reasons about - including the ones in a published latency figure.

Ordering is what makes the rest of the product possible. A backtest that replays events from several regions is a claim about sequence, and if the sequence is an artifact of clock drift, the research is measuring our infrastructure rather than the world. A cross-region latency figure is a subtraction between two clocks, so a figure computed against undisciplined ones is arithmetic on noise. When we say a record can be audited by the client, we are saying that the two timestamps on it were taken against the same reference - otherwise auditing it would be a ritual.

Here is the part most vendors leave ambiguous, and it is the most important sentence in this post. We stamp when we observed an event, not when it occurred. Those are different quantities, and the difference is not always small: an observation timestamp is a fact about our network, while an occurrence timestamp is a claim about someone else's system, and only one of them belongs in provenance as ours, which we can only ever infer. Presenting the first as the second is the industry's quietest inaccuracy, and it is tempting precisely because it makes a latency number look better.

So we publish the honest quantity and let the client do the arithmetic. The event time on a record is the instant the edge node heard it, against the network-wide reference; the serving node and its own timestamp are separate fields. Subtract them and you have the delivery interval for that record, computed by you, with no aggregate to trust. Where a source supplies its own occurrence time, that is carried as what it is - a value from a third party, with its own reliability - rather than blended into ours.

This also changes what a percentile means. A tail latency figure computed from the client's own records is a real distribution over the traffic that client actually receives, not a marketing summary over traffic they do not. It is a small design decision with an unusual consequence: our worst regions are visible to anyone who cares to measure them, which is uncomfortable and correct. A number you can only get from us is a number you should discount.

The operational half is unglamorous and continuous. Drift is monitored per node against the reference rather than assumed, a node whose offset exceeds tolerance is treated as degraded rather than as slightly off, and the tolerance itself is chosen to be an order of magnitude tighter than the intervals we quote. There is no version of this that is set up once. Clocks are a maintained property, like disk space, except that when they fail nothing crashes - the data just becomes subtly untrue, which is the more expensive failure.

Ordering is the thing you cannot add later. Everything else in the pipeline can be recomputed from what was stored: entities re-resolved, duplicates re-collapsed, models rerun over history. A record that was stamped by an undisciplined clock is unfixable, because the event time it should have carried was never captured. That is why the clock is the first thing in the path, not the last.