My scoreboard read 79.3% over nine units of weighted evidence, pooled across all six model arms. A hardcoded prior in the reporting script was discarding one arm's every success before the arithmetic ran. Matched to the seeds the writer used, the same pool reads 86.4% over seventeen. Neither figure was quotable, because no arm cleared the abstention floor. That is the finding.
Key facts. 79.3% over n=9 pooled across six arms, 86.4% over n=17 once the prior matched the producer. Neither was quotable: no arm cleared the abstention floor.
Last updated:
01The short version
On 2026-07-26 my scoreboard read 79.3%, pooled across every model arm. The number that matched the writer's own seeds was 86.4%. The cause was not bad data and not a bad model. The reporting layer had hardcoded a head start of 4 wins that the live system never granted, so an arm whose tally read 3.0 got 3.0 minus 4, clamped to zero, and every point the expensive model actually held was silently discarded from every report.
And the ending, which is the part I actually care about: after the fix, the number was better and still not reportable. Under the house floor of at least 5 real observations per arm, 0 of 6 arms were quotable either way. Both facts shipped in the same audit, in the same sentence.
02The Gambler and its scoreboard
Some vocabulary, because this workplace has named parts. The Gambler is the component that recommends which AI model gets each job. It learns by keeping score per model: each finished task lands as a win or a loss on that model's arm, the way a bookmaker adjusts odds after each race.
Two programs touch that scoreboard. The live system writes it, one task at a time. A separate reporting tool reads it and prints the headline win rates a human looks at. The writer and the reader were built weeks apart, by the same distracted manager. This story is about the day I learned they were keeping different scoreboards.
03Why did two reports disagree about the same model?
The live system seeds each arm deliberately. The cheap default model starts with a small benefit of the doubt, roughly four assumed wins baked into its prior, because it has a long track record here. The expensive model starts from nothing: one nominal win, one nominal loss. Zero head start. In Beta-prior terms, the seeds were (4,1) for the cheap model, (1,1) for the expensive one, and (4,2) or (2,1) for the domain-family arms. Every point the expensive model held above that prior came from offline eval runs carried at half weight, and the state file tags those arms seeded.
The reporting tool assumed one rulebook for everybody. Hardcoded inside it: every arm begins with the same four-win head start, so to recover the real wins, subtract four. From every arm. Including the one that was never given them.
04Three minus four is zero
The expensive model's arm sat at a tally of 3.0. The reader subtracted its imagined 4 and got minus 1. Code does not print negative wins, so it took max(0, minus 1) and printed zero.
That single clamp is what makes this failure shape nasty. A negative number in a report would have screamed. Zero looks like an answer. Every success the expensive model earned was erased before a human ever saw it, in every report, for as long as both programs existed, and nothing anywhere looked broken. One detail worth being precise about: the source records the arm at a tally of 3.0, which includes its one-win seed, so the offline evaluation runs at half weight underneath were 2. The eraser did not care either way. It erased everything.
05Caught mid-write
The date is 2026-07-26 and I am writing the audit that is supposed to be the honest one, the document produced after an adversarial re-audit had already embarrassed my first verifier. The headline was typed: 79.3%, interval 55.9 to 95.2, nine units of evidence. The house rule says every headline number gets re-derived at write time, from the producer's own files, not carried forward from earlier in the session. So, grudgingly, I re-derived it.
With the reader's priors matched to the writer's, the same scoreboard read 86.4%, interval 69.6 to 97.0, seventeen. Both figures are pooled across all six arms; neither is one model's record. And the unit needs saying plainly: every point on those arms came from offline eval runs carried at half weight, so seventeen is weighted evidence, not seventeen live jobs. Eight units had been vanishing from every report. Notice the direction: the bug did not flatter the system. It made the expensive employee look worse than he was. Silent data discard has no preferred direction, which is exactly why you cannot wait for a suspiciously good number to go looking for it.
06Better, and still not quotable
The corrected figure reproduces from the audit's own line: 15.0 successes and 2.0 failures, all of it offline eval evidence at half weight, plus the one legitimate prior. (15 + 4) / (17 + 5) = 86.4%, and the prior supplies 21% of that numerator. Which means even the fixed number is one-fifth assumption by mass, pooled across six arms that are not really comparable to each other.
And then the abstention rule got its say. An arm needs at least 5 real observations before its win rate means anything here. MIN_N = 5, no exceptions for numbers that flatter. On 2026-07-26, 0 of 6 arms had cleared that floor. So the correction produced a strange artifact: a number that got better AND stayed unreportable. The audit published it anyway, wearing its caveat in the same sentence: 86.4%, and no single arm is quotable yet. I think that sentence, the improvement and the abstention riding together, is the most honest thing the system has ever printed.
07What this buys you
Two rules came out of this, and they generalize past AI systems to any two programs exchanging data through a file. First: the reader must accept what the writer actually writes. Same field names, same units, same priors. A missing or mismatched key is UNKNOWN, never coerced to zero, because a coerced zero is indistinguishable from a real one.
Second: any number bound for a headline gets re-derived by a script at write time, and that script names the file and the key its data came from. The reporting tool in this story still does neither. A hand-carried number has no way to notice it is wrong. Mine was wrong from the moment the reporting tool existed until the same morning it was first quoted, in the unflattering direction, and it took a house rule, not vigilance, to catch it three sentences before publication.
08How every number here was derived
Both source files are append-only, so re-running these will give larger denominators than the ones quoted.
# every figure in this post, re-derivable from the audit verdict grep -n "79.3\|86.4\|Beta(4,1)\|a=3.0" ~/.claude/AIOS-AUDIT-2026-07-26/VERDICT-v2.md # the floor that erased the wins python3 -c "print(max(0, 3.0 - 4))" # 0 # the corrected headline: 15 live successes + the (4,1) prior over n=17 + 5 python3 -c "print(round((15.0 + 4) / (17 + 4 + 1) * 100, 1))" # 86.4 # share of the numerator supplied by prior, not evidence python3 -c "print(round(4 / (15.0 + 4) * 100))" # 21 # the abstention: no arm cleared the n>=5 floor grep -n "None has cleared" ~/.claude/AIOS-AUDIT-2026-07-26/VERDICT-v2.md