Deduplication

Deduplication removes redundant records so that one real-world event shows up once rather than several times. It sounds trivial and is not: the same news story is syndicated across dozens of outlets with small edits, the same listing is re-posted, the same observation arrives twice from an at-least-once stream. Exact-match dedup catches the identical copies; the harder work is near-duplicate detection across small variations.

It matters because duplication inflates counts and biases any aggregate built on top - a signal that double-counts a syndicated story overstates its strength. Techniques range from hashing for exact matches to fuzzy similarity and clustering for near ones, and the line between deduplication and entity resolution blurs as the matching gets softer. In a real-time pipeline it has to happen inline, fast enough not to add latency.