Data Observability

Data observability is the practice of monitoring a dataset rather than the machines carrying it. Infrastructure monitoring answers whether the job ran; observability answers whether what the job produced is right. The four things worth watching are freshness (is the newest record recent), volume (did the expected quantity arrive), distribution (do the values still look like themselves), and schema (did the shape change) - plus the lineage needed to trace an anomaly back to its origin.

The failure it exists to catch is silence, not error. A source that starts returning an empty but well-formed response raises no alarm anywhere: the request succeeds, the pipeline runs, the dashboard is green, and the data quietly stops. Because the symptom is an absence, it is invisible to error-rate monitoring and often surfaces only when someone notices a number that stopped moving. Treating a quiet source as a defect rather than a lull is what turns data quality from a periodic audit into a live property, and it is why freshness is worth publishing rather than assuming.