The check that stopped us publishing
Our own safety check refused to update the site, and it held the front page on the previous day's race for most of a Saturday. It was wrong. We kept it anyway, and fixed the part that was actually broken.
What the check is for
Every number on this site has to be registered before it can be published. Not reviewed afterwards, not caught in proofreading — registered, in advance, with its definition and its source attached. If a figure is not in that register it cannot reach a page, and the deploy is refused rather than the figure being quietly dropped.
Some figures in that register are deliberately marked not for publication. They exist because we computed them and want the record to show we did, but we do not consider them defensible enough to put in front of a reader. So one of the checks takes each of those withheld figures and searches the finished pages for it. If it finds one, nothing deploys.
That is a blunt instrument, and the bluntness is the point. A check that requires judgement to pass is a check you will eventually talk yourself past.
What went wrong
The check looks for the digits. It has no way of knowing which quantity a number on the page refers to, so it treats any appearance of the right digits as a leak.
On Saturday our public ledger — the running count of reads we have published before the off — reached a total that happened to be exactly equal to one of the withheld internal figures. Two completely unrelated quantities, the same integer. The check found the digits on the homepage, could not tell that they belonged to the ledger rather than to the withheld figure, and refused the whole deploy.
Nothing was leaking. The withheld figure was not on the page and never had been. The only thing on the page was a count of our own published reads, labelled as exactly that. But the site stayed on the previous day's race while an entire card was run.
Why we did not just switch it off
The obvious move, with the afternoon disappearing, is to skip the check once. We have seen enough systems to know where that goes. A control that gets bypassed the first time it is inconvenient is not a control, it is a formality, and the leak it was built to catch will arrive on a day when nobody is paying attention.
So we narrowed it instead. A withheld figure now only counts as published if the digits appear near the words that describe what that figure actually measures. A real leak reads like the thing it is — the number sits beside its own label. A coincidence does not.
Then we did the part that matters more than the fix. We put a genuine leak back into the page on purpose, ran the check, and confirmed it still failed. A repaired check that can no longer fire is worse than the false alarm it replaced, because it is silent in both directions and you cannot tell the difference from the outside. Clean page: passed. Leak deliberately inserted: failed, and it named both the figure and the page. Page restored: passed.
The part worth keeping
There is a version of this story where nobody outside ever hears about it. The gate was ours, the fix was ours, the only visible symptom was a stale date on a page, and by the time anyone looked it was all working again.
We are writing it down because the thing we are actually selling is that our record is checkable, and a record is only checkable if the controls around it are real enough to be inconvenient. Ours cost us the better part of a race day. That is the correct price, and the alternative — a gate that never gets in the way — is not a cheaper version of the same thing. It is a different thing that looks similar from a distance.
Two other failures surfaced in the same deploy, and they were more ordinary: a market reading had never been derived, so a label on the card was wrong, and a credential for one of our own data stores had gone missing. Both were real. Both were caught by the same gate. Of the three reasons the site refused to publish that day, only one was the gate being wrong.
Check today’s races
See how Elite Pass reads today’s eligible races.
Related research
More notes
- What two months of testing actually taught usThree of the first five beliefs we tested did not survive. Here is what lived, what died, and what changed about the question we were asking.
- What Elite Pass Gold is actually trying to solveOdds already answer the likelihood question, and answer it well. We are trying to answer a different one: how strongly does the pre-race picture agree with itself?
- How should a horse racing system actually be tested?Twelve steps, in order, for testing a racing idea properly. Written by someone who has run 227 of them and cleared none.
Elite Pass publishes research, not guaranteed outcomes. Findings are measured against the market and remain subject to replication and sample size. The evidence room and the public ledger carry the full record. All insights.