The backtest that knew too much

A research result that is too good is usually not a discovery, and the reason is almost never the code. Look-ahead bias - a test using information that did not exist at the moment it simulates - has a reputation as a beginner's mistake, which is why it survives so comfortably in mature research stacks. The obvious version, an off-by-one on a timestamp, gets caught in review. The version that ships to production arrives inside the dataset, wearing the vendor's authority.

It enters through three doors. The first is silent revision: a vendor corrects a value and overwrites it in place, so a query for what was known on a past date returns today's better answer. The second is period keying, where a record is stamped with the quarter it describes rather than the date it became knowable, which hands your test a filing weeks before anyone could have read it. The third is the surviving universe, where the names that failed have been quietly dropped, and survivorship bias makes the past look like a market with no losers in it.

None of these announce themselves, and all three flatter the result. That is what makes the bug expensive: an inflated backtest does not look broken, it looks promising. It clears review because reviewers examine the logic rather than the provenance of the numbers, and it fails only when capital is deployed into a world that has not happened yet. By then the failure is indistinguishable from ordinary decay, which is why the same mistake can be made twice by the same team on two different datasets.

There is a test that takes an afternoon and settles it for any vendor, including us. Run the same historical query twice, as of two different dates, and diff the results for a window that both cover. If the older window came back different the second time, the archive is not point-in-time - it is a live table with a date filter, and every backtest run against it has been reading corrected values. Ask, too, what happens to a record that is superseded: an archive that answers by naming the version and the moment each was known has been designed for this, and one that answers by describing a data-quality process has not.

The deeper reason this problem belongs to the data layer rather than to the researcher is that it is only fixable at the moment of writing. Once a value has been overwritten, the knowledge of when the original was true is gone, and no amount of care downstream reconstructs it. That is why records carry the event time they were observed at, why provenance travels with them rather than living in a separate report, and why we treat a correction as a new record with its own timestamp instead of an edit to an old one. The archive is append-only because history is.

Worth being precise about what point-in-time does and does not buy you. It does not make a signal real, and it will not save research from overfitting, which is a much more common cause of a strategy that works until it is funded - nor will it fix a signal that was never there. What it buys is that the test is answering the question you asked. Removing look-ahead does not improve a result; it makes the result mean something, and a great many promising numbers get smaller the first time they are computed honestly.

Two smaller habits close most of the remaining gap. Compute features against the moment information became available rather than the period it describes, so a revision three weeks later is a new observation instead of a retroactive improvement - the discipline behind sound feature engineering. And keep the delisted, the withdrawn, and the corrected in the universe with the dates they left it, because a dataset that only contains what worked out is a description of the past written by the winners.

The uncomfortable part of all this is that the more rigorous your history, the worse your research looks, at least at first. That is the correct direction. A backtest is a hypothesis test, not a promise, and a hypothesis tested against a world that already knew the answer has not been tested at all.