Worse Than a Coin Flip

01The answer, first

On 2026-07-27 UTC, day 30 of building an operating system around a terminal AI, I ran the first benchmark I did not write against the Gambler, the component that recommends which model gets each job. Across the full 36,497 rows, a router that assigns models at random scores 0.5202 and one that always picks the cheapest model scores 0.3061. Replayed with exploration off over a 5,000 row subsample, my live configuration scored 0.3108, which on that subsample is exactly the always-cheap score.

Every earlier verdict on this system was produced inside my own workspace, and none of them noticed. That is the whole post. Self-assessment is not external validity, and in my case the gap between the two was the gap between fine and worse than a coin flip.

02A month of grades from inside the building

The Gambler had been reviewed constantly. The loudest review was a 44 agent self-run campaign that graded the whole system world class on 2 of 10 dimensions, competitive on 8, behind on 0. It was honest work, and I want to be fair to it: it measured coverage, whether a component existed for each concern a serious system should have. Coverage was real.

But every one of those graders was built by me, ran on my task mix, and used my definition of success. The audit that followed put it plainly: that verdict is evidence about coverage, not an independent ranking, and no external benchmark had ever been run against the system. Homework graded by the family.

03Why grade your own system with a benchmark you did not write?

RouterBench is a public benchmark of 401,467 precomputed inference outcomes, 36,497 rows scored against each of 11 models. You replay your routing policy over it, and it scores the quality and the cost of the assignments the policy makes. Nobody who built it has ever seen my system, which is precisely its value.

The three numbers again, because they are the post. On the full 36,497 rows: random assignment 0.5202, always-cheapest 0.3061. On a 5,000 row subsample with exploration off: my shipped config 0.3108, which is the always-cheap score on that subsample, because the policy pulled the cheap arm on every one of the 5,000 rows. Not near it. Equal to it, and equal for a reason.

04I misread it on the first pass

My first report on the exam said we passed: 0.7699, comfortably above the coin. The number was real and the verdict was wrong. I had added 5 percent forced exploration to the simulation to keep the policy from sticking, and then failed to account for it. The pass mark belonged to my test harness, not my router.

This is the part I would most like you to keep. Even the external exam got graded by the family on the first pass. The correction sits in the same written report as the claim, and the corrected number is the one in bold.

05The absorbing proof

Why did a tuned policy collapse to always-cheap? The mechanics are short. At task zero, no model has the 5 observations my abstention gate requires before it will trust a score. So the policy falls back to the default arm, the cheap model. Only that arm accumulates observations, so only that arm can ever pass the gate, and every other model is scored minus infinity forever. Over 5,000 simulated tasks the policy chose one distinct model 5,000 times. One honest caveat the artefact itself insists on: in the simulation the unrated fallback is hardcoded to the cheapest arm, while the live system falls back to the mid tier, so the live version of this trap is more expensive and better than the one modelled here.

The confirming detail: the entire tuning grid produced byte identical output at zero exploration. Parameters I had argued with myself about for weeks could not have mattered. The abstention gate is a good rule on its own. Combined with zero exploration, it is a trap that closes on the first arm touched.

06Spending the refutation

A refutation is only depressing until you spend it. Retuning against the exam showed the pessimism quantile I had fussed over was a red herring, with efficiency 11.63x at q=0.10 versus 11.24x at q=0.75, barely a difference, and that the cost term was the real lever. Every row of that sweep was measured with exploration on, so none of them is the config that shipped. At cost weight 0 the policy holds 90.0 percent of always-expensive quality at 76.1 percent of its cost. At cost weight 100 it holds 77.0 percent of the quality at 7.4 percent of the cost. At 1000 it holds 75.7 percent at 6.5 percent, the point where the cost term stops paying. The setting was recommended, not adopted: the live decision rule still carries no cost term.

Exploration was armed in the same push, with a minimum detectable effect calculation attached, so the next verdict arrives with a sample size instead of a feeling.

07Self-assessment is not external validity

Every internal check shares its author's assumptions: the task mix, the definition of success, the simulation code, the blind spots. A benchmark you did not write is a different instrument, and it disagrees with you in ways your own tests cannot, which is the entire point of it.

The audit written the day before the exam said flatly that no external benchmark had ever been run against this system. This was the first, it refuted the live config, and no second one has run since. So the standing rule here is now written down: every comparative claim about this system is self-assessed until a second instrument reports. If your system has never taken an exam someone else wrote, you do not know its grade. You know its homework average.

08How every number here was derived

Both source files are append-only, so re-running these will give larger denominators than the ones quoted.

# 0.3108, 0.7699 vs 0.5202, the 5% artifact, always-cheap equality, absorbing proof, retune table
sed -n '80,102p' ~/.claude/AIOS-POST-V3-COMPLETION-2026-07-28.md
# 405,000 precomputed outcomes, 11 models, first external measurement
grep -n -B2 -A6 "RouterBench, and the refutation" ~/.claude/HANDOFF-aios-2026-07-28-30h-complete.md
# 44-agent campaign was self-run; no external benchmark ever run before this
grep -n -A4 "world-class 2/10" ~/.claude/AIOS-AUDIT-2026-07-26/VERDICT-v2.md
# day one 2026-06-27 (2026-07-28 = 31 elapsed days, the \"thirty days\" in the hook)
grep -n "Day one" ~/.claude/HANDOFF-aios-2026-08-01-MASTER.md
# the finding as carried in the master handover (5,000 rows, 1 arm)
grep -n "RouterBench" ~/.claude/HANDOFF-aios-2026-08-01-MASTER.md