We label the noise, we do not delete it
Filtering on a customer's behalf sounds like a service. It is an invisible threshold applied to decisions you own - so we publish numbers instead.
The most requested feature in real-time data is less of it. Every team that has integrated a high-volume stream has spent a quarter building the layer that decides what to ignore, and every one of them has asked whether the vendor could just do that upstream. It is a reasonable request and we decline it, for a reason that takes a paragraph to explain and a career to unlearn if you get it wrong.
A filter is a threshold, and a threshold is a decision about what a false negative costs. That number is different for a macro desk and a credit desk, different again for a model being grounded on current events, and it is not a number the vendor holds. When we delete a record because our models judged it unimportant, we have made an unrecoverable choice on behalf of somebody who bears a consequence we do not see. Worse, the choice is invisible: a client can audit what arrived, but nobody can audit what was quietly withheld. There is no metric for a signal you never received.
The strongest version of the counter-argument deserves stating, because it is mostly right. Most teams do not want to tune anything. Defaults are a real product, and a vendor that ships none is externalizing work while calling it flexibility. A subscription that arrives at a sane volume on day one is worth more to most buyers than a philosophically pure one that turns data quality into a configuration exercise that requires two weeks of threshold-setting before it is usable.
So the resolution is where the filter runs, not whether it exists. Every record carries urgency, authenticity, and data quality as values, and a subscription can be defined against them, with defaults we recommend and publish. The difference is that the filter is a client-side parameter over a complete stream rather than a deletion upstream: change the threshold and yesterday's records are still there to be re-read, which means a team can measure what a stricter setting would have cost them instead of guessing. Raising the signal-to-noise ratio is our job; deciding where the line falls is theirs.
This puts an obligation on us that filtering would have hidden, and it is the part of the design that costs real engineering. A number a client sets a threshold against has to mean the same thing next quarter as it does today. If a retrained model shifts the distribution of an urgency value, every downstream threshold silently changes meaning - a client who set 0.8 to catch the top few percent is now catching the top fifteen, and nothing in their system reports an error. That is model drift with a commercial blast radius, and the honest handling of it looks like schema evolution: evaluation versions are part of the contract, changes are announced with a horizon rather than deployed silently, and history remains readable under the version it was written with.
A second obligation is that the values have to be worth the space they occupy. Three numbers that all correlate are one number wearing a costume. Urgency, authenticity, and quality earn separate fields because they fail independently: a real event reported sloppily is authentic and low quality; a well-formed record from a source imitating a credible one is high quality and inauthentic, which is exactly what entity resolution and corroboration exist to catch; a perfectly good record about something that does not matter to anyone is both and still not urgent. Collapsing them into a single confidence figure would be easier to explain and would destroy exactly the distinction a desk needs.
The design also answers a question we get from research teams, which is what to do with the low end of the distribution. The answer is that it is data too. Records our models judged unimportant are still evidence of what was happening and of what our network saw, and teams doing feature engineering or measuring how a story propagated need the tail as much as the peak. A vendor that deleted it has made that work impossible after the fact, which is a strange thing to do to a dataset whose value is supposed to compound.
The short version: we will happily reduce the volume you receive, and we will not decide what you are allowed to have received. Those sound similar and they are the difference between a tool and a verdict - which is the same reason every record carries its provenance instead of a grade.