How should a horse racing system actually be tested?

1. Define the idea before you test it

In writing, in advance, specific enough that somebody else could apply it without asking you a question. "Well-drawn horses in big-field sprints" is not an idea, it is a mood. Which draws, which field sizes, which distances, which going, which race classes.

The reason is not tidiness. Every detail you leave open is a detail you will close later, after seeing the data, in whichever direction helps.

2. Define which races qualify

Then count them before you look at any result. If the number of qualifying races changes during the test, the test has changed, and you should be able to say exactly why.

3. Decide what missing data means, in advance

Non-runners, void races, missing going, abandoned meetings, races with no exchange price. Decide the rule beforehand. Deciding afterwards is how a population quietly becomes the subset where the idea worked.

Our own rule is that missing is never treated as zero and never silently dropped. It is recorded as missing and it stays in the denominator.

4. Choose the benchmark before you know the answer

The right comparison for almost any racing idea is the market price, not zero and not a coin. Beating random is meaningless. The question is whether the idea adds anything to what the price already says.

We use Betfair Starting Price because its built-in margin is 0.2% against 19.2% for bookmaker starting price, so it is much closer to a clean probability. On our paired comparison across 1,997 races its calibration slope was 0.974, interval 0.925 to 1.023, an interval that contains 1.000.

That makes BSP a good benchmark. It does not make it unbeatable, and we are not claiming that.

5. Separate the races you develop on from the races you test on

Split by time, not at random. Build on the earlier period, test on the later one, and do not look at the later one until you have finished deciding.

If you look, it is no longer a test. Lo and MacKinlay's 1990 result is exactly this: reusing one dataset to both discover and confirm a pattern biases the confirmation, and the bias is not small.

6. Timestamp prospective selections

The strongest evidence a racing idea can produce is a record of what it said before the races were run. Everything else is an argument about a dataset.

Timestamp the selection and the price you could actually have taken. A price read after the race has already been changed by the result.

7. Record every qualifying race

Not the ones you noticed, not the ones you had a bet on. Every race that met the rule, including the days you were not paying attention.

8. Do not delete the losers

The temptation is never to delete a losing bet. It is to decide, afterwards, that a race did not really qualify. That is the same act with better manners.

9. Count how many things you tried

This is the step almost nobody takes and it is the one that decides whether the result means anything.

Bailey, Borwein, Lopez de Prado and Zhu put it plainly in the Notices of the American Mathematical Society in 2014: high simulated performance is easily achievable after backtesting a relatively small number of alternative configurations, and the more you try, the more likely the result is overfit. Their practical observation is that the number tried is almost never reported, so a reader cannot judge how overfit a result is.

Halbert White's Reality Check formalised the correction. Harvey, Liu and Zhu argued that in a field which has tested very many candidate effects, the conventional significance threshold is simply too weak.

A rule that survived one attempt and a rule that survived four hundred are not the same evidence, even when the numbers look identical.

10. Compare against the market, not against nothing

Ranking horses well is not an edge. Our ratings study is the cleanest example we have: across 30,088 races and 298,444 runners, the top-rated horse won at 0.998 of the rate its own price implied. The rating was good. It was also already in the price.

11. Allow the answer to be no edge

If the design cannot return no, it is not a test. Decide in advance what result would make you abandon the idea, and write that down too.

12. Keep watching after you have decided

A prospective record does not finish. Effects decay, markets adjust, and a rule that worked for a season is a rule that worked for a season.

Judging a probability rather than a tip

If your idea outputs probabilities rather than selections, there is a better way to score it than counting winners.

Glenn Brier proposed the original approach in Monthly Weather Review in 1950 for weather forecasts: score the probability you gave against what happened, across many forecasts. Tilmann Gneiting and Adrian Raftery set out the modern treatment in the Journal of the American Statistical Association in 2007, on what makes a scoring rule proper, meaning one a forecaster cannot improve by reporting something other than their honest belief.

This matters in racing because it separates two things that get confused constantly. A tipster is judged on winners. A probability is judged on calibration, and a forecast that says thirty percent should be right about thirty percent of the time.

Why we hold ourselves to it

Because we have watched our own best ideas die at step ten, repeatedly, and it would have been very easy not to notice.

Zero systems cleared is an uncomfortable number to publish. It is also the only honest denominator we have, and a record without a denominator cannot be read at all.

The system audit is the running list. The evidence room holds the record. What is Betfair Starting Price explains the benchmark.

See how the same discipline is applied to today's card.