Two buyers, one record
A desk wants one event as early as possible. A training run wants years of consistent history. One design decision lets the same records serve both.
The two buyers for real-world data want almost opposite things. A trading desk wants a single event delivered before anyone else has it, evaluated well enough that nobody has to read it first, and it will trade away completeness for speed every time. A team building models wants enormous, consistently structured training data with a history that does not misrepresent what was knowable at each moment, and it will trade away speed for consistency every time. Most vendors resolve this by building two products, and then the buyer pays for the seam.
The seam is expensive in a way that does not show up in the contract. When the live stream and the historical extract are assembled by different systems, they diverge - different deduplication, different entity mapping, different enrichment vintages - and the divergence lands on the one team least equipped to notice it. Research validates a signal on the archive, production runs on the stream, and the difference between them shows up as unexplained decay that gets attributed to the market rather than to the plumbing. It is a data-engineering failure that presents as an alpha problem.
The design decision that removes the seam is to do the expensive work while the record is in motion, and to split only at delivery. Entities are resolved, duplicates are collapsed, and urgency, authenticity, and data quality are evaluated once, on the way through the network, before anything is written down. What we call the archive is then the same records at a different distance: a streaming subscriber sees them as they pass, a REST client sees them as point-in-time history, and both are reading the identical values because nothing was recomputed.
That constraint is unforgiving on the evaluation layer, and it is worth being explicit about the cost. A model that runs inline has a budget measured in the milliseconds a record spends in flight, which is a tail latency constraint rather than an average one, which rules out approaches that would be perfectly reasonable in a nightly job. We accept that: at these volumes a slow model is functionally a wrong model, because a queue that grows faster than it drains produces stale answers regardless of how good they were when they were computed. Being fast enough to sit in the path is a design requirement, not an optimization we got around to.
The payoff is that a backtest becomes predictive of production rather than merely encouraging. If the evaluation values on a record from eighteen months ago were assigned by the same models, at the same point in the path, as the ones arriving now, then a threshold tested on history is a threshold that means the same thing live. This is also the practical reason we care so much about look-ahead bias: the whole benefit collapses if the archive quietly improved the past while nobody was watching.
There is a version of this argument that goes too far, so here is the limit. The two buyers still consume differently, and pretending otherwise produces a worse product for both. A desk wants a narrow, high-urgency subscription pushed the instant it is observed; a training pipeline wants bulk extracts with reproducible boundaries and no surprises about what a rerun will return. Serving them from one set of records is not the same as serving them through one interface with one shape - it means the guarantees hold across both, while the access patterns are allowed to differ.
It also explains a choice that surprises people: the evaluation values are exposed as numbers rather than applied as a filter. A desk and a training run do not want the same threshold, and neither of them wants ours. If we deleted what we judged unimportant, one buyer would lose the recall it needs and the other would inherit a decision it never agreed to, made by a party that does not carry the consequence.
One network, one evaluation, two ways to read it. The architecture is the answer to a commercial question - whether a desk and a model can be served without building the same company twice - and the reason we can answer it at all is that nothing in the path is subcontracted to someone else's pipeline.