Why so many horse racing systems look brilliant in backtests

The seven ways a backtest flatters you

These are not exotic failures. Every one of them is easy to commit while being careful and meaning well.

Searching, then testing on the same data. You look through ten years of racing, notice that a particular kind of horse did well, and then check whether that kind of horse did well. It did. That is not a test, it is the same observation twice. Andrew Lo and Craig MacKinlay set this out formally for asset pricing in 1990: reusing one dataset to both discover and confirm a pattern biases the confirmation.

Trying many rules and reporting one. Twenty filters, each with a one-in-twenty chance of looking good by luck alone, will hand you roughly one winner. Halbert White's 2000 paper in Econometrica gave this a proper treatment: a test applied after a search has to account for the search, or its significance is fiction.

Tuning until it works. Field size of eight or more becomes seven or more becomes seven to twelve. Each nudge is defensible. Together they are a search, and nobody counts them as one.

Look-ahead. Using anything the punter would not have had at the time. Final starting price is the classic. So is a going description updated after the meeting, or an official rating revised the following week.

Survivorship. Testing on the meetings, seasons or tracks that made it into your file, when the ones that did not are missing for a reason.

Silent exclusions. Non-runners, void races, meetings abandoned halfway. Each removal is small and reasonable and the population quietly stops being the population you claim to have tested.

Telling the story afterwards. The rule works, so you explain why it works. The explanation is invented after the result and adds no evidence whatever, but it makes the finding feel earned.

What the research actually says

This is not a racing problem. It is a statistics problem that racing happens to be an excellent place to commit.

In 2014, David Bailey, Jonathan Borwein, Marcos Lopez de Prado and Qiji Jim Zhu published a paper in the Notices of the American Mathematical Society with the unusually blunt title Pseudo-Mathematics and Financial Charlatanism. Their central claim is that high simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations, and that the more configurations are tried, the greater the probability the backtest is overfit. Their practical point is sharper still: because the number tried is rarely reported, a reader usually cannot evaluate how overfit a result is.

Campbell Harvey, Yan Liu and Heqing Zhu made a related argument about finance research in the Review of Financial Studies, that when a literature has tested very many candidate effects, a conventional significance threshold is simply too weak to sort the real ones from the lucky ones. John Ioannidis made the general version of the case in PLoS Medicine in 2005: the reliability of a finding depends on how many hypotheses were in play, not only on the finding's own p-value.

None of these papers is about horse racing, and we are not going to pretend a finance result proves a racing result. What transfers is the statistical problem, which does not care what the rows in the dataset represent.

An illustrative example

Invented for illustration, and typical of the shape.

Somebody notices that second favourites in small-field handicaps on soft ground did unusually well over five seasons. The sample is a few hundred races, the return looks strong, and there is a story to go with it: soft ground exaggerates stamina differences and small fields reduce interference.

What is not in the write-up is that soft ground was tried after good ground and heavy ground did nothing. That the field-size cut moved twice. That handicaps were chosen because the first pass across all race types was flat. That second favourite was reached after favourite and third favourite were both disappointing.

Nobody lied. Each decision was made for a reason. But the finding is the survivor of a search that was never counted, and the write-up describes only the survivor.

Why we publish the failures

The number that matters in our audit is not 227. It is 0, and it is the least flattering figure we own.

We publish it because of exactly the problem above. A record that shows only the ideas that worked is uninterpretable: you cannot tell whether it represents one success in three attempts or one in three hundred. The denominator is the evidence. Without it a good result is just a survivor with a story.

A racing idea should have to earn the right to be believed.

That is also why our daily reads are timestamped and frozen before racing. An idea that can be revised after the result is not being tested. It is being narrated.

The system audit lists what we have examined. The evidence room holds the record. What price control did to our best looking ideas is the specific case of our own most promising work dying this way.

See how the same discipline is applied to today's card.