Two programs shared one scoreboard file. The one that writes it gives each model a different head start. The one that reads it had a single head start of 4 hardcoded inside. So an arm sitting at 3.0 became 3.0 minus 4, clamped to zero, and every point the expensive model had earned was discarded before a human saw it. A rule requiring headline numbers to be re-derived at write time caught it three sentences before publication. The corrected figure was better and still not quotable.
Key facts. Report said 79.3% over n=9. Re-derived with matching seeds: 86.4% over n=17. Neither was quotable: 0 of 6 arms had cleared the minimum sample floor.
Last updated:
01My scoreboard printed zero wins for a model that had earned them
One part of my AI coding system recommends which model gets each job. It learns by keeping score: each finished task lands as a win or a loss on that model's tally, the way a bookmaker adjusts odds after each race.
Two programs touch that scoreboard. The live system writes it, one task at a time. A separate reporting tool reads it and prints the win rates a human looks at. They were built weeks apart, by me.
They were keeping different scoreboards.
The expensive model's arm sat at a tally of 3.0. The reporting tool subtracted a head start of 4 that the live system had never granted it, got minus 1, and printed max(0, minus 1). Zero.
That clamp is what makes the failure shape nasty. A negative number in a report screams. Zero looks like an answer.
stored tally for the expensive model: 3.0head start the reader subtracts: -4-------1.0printed as max(0, -1.0): 0head start the writer actually gave: 1
02The two programs kept different scoreboards
Seeding an arm is deliberate. A model with a long track record here starts with a small benefit of the doubt, so its first unlucky task does not tank it. A model with no track record starts from nothing.
So the writer gave the cheap default a head start of roughly four assumed wins, and gave the expensive model one nominal win and one nominal loss. Zero head start. The domain-specific arms got their own values again.
The reader assumed one rulebook for everybody: every arm begins with the same four-win head start, so to recover real wins, subtract four. From every arm. Including the ones that were never given them.
Neither program is wrong on its own terms. The seeding scheme is standard practice, and so is the subtraction. The defect is that one of them has a constant baked in where the other has a per-arm value, and nothing forced them to agree.
03A standing rule caught it. Vigilance would not have.
On 2026-07-26 I was writing an audit that was supposed to be the honest one, produced after an earlier verifier had already been embarrassed. The headline was typed: 79.3 percent, interval 55.9 to 95.2, nine units of evidence.
The house rule is absolute: every headline number gets re-derived at write time, from the file that produced it, never carried forward from earlier in the session. So I re-derived it.
With the reader's assumptions matched to the writer's, the same scoreboard read 86.4 percent, interval 69.6 to 97.0, seventeen units. Eight units of evidence had been vanishing from every report since both programs existed.
Notice the direction. The bug did not flatter the system. It made the expensive model look worse than it was. Silent data discard has no preferred direction, which is exactly why you cannot wait for a suspiciously good number before going looking.
04The corrected number was better and still not publishable
Reproducing the corrected figure takes one line: 15.0 successes and 2.0 failures, plus the one legitimate head start. (15 + 4) divided by (17 + 5) is 86.4 percent.
Which means the fixed number is still about a fifth assumption by mass, pooled across six arms that are not really comparable to each other.
Then the abstention rule got its say. Before an arm's win rate means anything here it needs at least 5 real observations. On that date, 0 of 6 arms had cleared the floor. Not the expensive model, and not the cheap one with its long track record either.
So the correction produced a strange artefact: a number that got better and stayed unreportable. The audit published both in the same sentence. I think that is the most honest thing the system has printed.
One unit note, because it changes what the figure means. Every point on those arms came from offline evaluation runs carried at half weight. Seventeen is weighted evidence, not seventeen live jobs.
05How to check for this in your own system
This generalises well past model routing. It applies to any two programs exchanging data through a file, which is most systems.
The rule underneath is that the reader must accept what the writer actually writes. Same field names, same units, same seeds. A missing or mismatched key is unknown, never coerced to zero, because a coerced zero is indistinguishable from a real one.
06What this buys, stated plainly
A hand-carried number cannot notice it is wrong. Mine was wrong from the moment the reporting tool existed until the morning it was first quoted, in the unflattering direction. A standing rule caught it. Attention never would have.
None of the machinery here is novel and it would be strange to claim otherwise. Seeding an arm with assumed wins is a standard technique, and a minimum-observation floor is the same guard every A/B testing tool ships to stop you reading a barely-sampled result as a verdict.
What this adds is only what happens when the writer and the reader hold different seeds for the same arm, and how cheaply a re-derivation rule catches it.
07How every number here was derived
These commands read live state, so re-running them will give different numbers from the ones quoted here, usually larger. That is the point of printing the commands rather than only the conclusions.
# every figure in this post, re-derivable from the audit verdict
grep -n "79.3\|86.4\|Beta(4,1)\|a=3.0" ~/.claude/AIOS-AUDIT-2026-07-26/VERDICT-v2.md
# the clamp that erased the wins
python3 -c "print(max(0, 3.0 - 4))" # 0
# the corrected headline: 15 successes + the legitimate prior over n=17 + 5
python3 -c "print(round((15.0 + 4) / (17 + 4 + 1) * 100, 1))" # 86.4
# share of that numerator supplied by prior rather than evidence
python3 -c "print(round(4 / (15.0 + 4) * 100))" # 21
# the seeds the writer actually stores, per arm
jq -c '.arms | to_entries[] | {(.key): {a: .value.a, b: .value.b, seeded: .value.seeded}}' \
~/.sgnk/state/routing-bandit.json
# the floor, and how many arms have cleared it
grep -n "MIN_N" ~/.sgnk/bin/routing-bandit-decide.sh