Tail Latency

Tail latency is the delay experienced by the slowest requests in a distribution, usually quoted as p95, p99, or p99.9. It matters because the median is a poor description of a system anyone depends on: a pipeline with a 200-millisecond median and a four-second p99 is a four-second pipeline for the one update in a hundred that decides something. Averages are worse still, since a small number of very slow records can hide inside one entirely.

The tail is also where most real failures live - a retried connection, a garbage-collection pause, a region rerouting - and it compounds through a chain, because a request that touches several stages inherits the tail of each. This is why an honest latency figure is a distribution rather than a single number, and why a figure computed from the event time and serving node on each record is worth more than an aggregate in a datasheet: you can compute your own percentiles instead of accepting ours.