For thirty days every verdict on my model router came from a test I wrote. Then I replayed the shipped policy over RouterBench, a public benchmark I had no hand in. On a 5,000 row sample with exploration off it picked one distinct model 5,000 times, and every setting in the tuning grid produced byte-identical output. It was not routing. It was repeating. The cause is a good rule and a missing one: an abstention gate that needs 5 observations, and no exploration to ever produce them.
Key facts. 5,000 rows replayed, 1 distinct model chosen. Every setting in the tuning grid produced identical output.
Last updated:
01My router stopped choosing, and thirty days of my own tests said it was fine
One part of my AI coding system recommends which model should run each job. Cheap model for small work, expensive model for hard work. That is its entire purpose.
On 2026-07-27 I replayed the shipped configuration over RouterBench, a public benchmark of precomputed inference outcomes. It holds 401,467 results: 36,497 tasks scored against each of 11 models, so you replay your routing policy over it and it reports the quality and the cost of every assignment that policy makes. None of it is mine.
The shipped policy sent every row to the same model. Not most of them. All 5,000.
Whatever you think of how a benchmark scores an assignment, a policy that returns one answer to 5,000 different questions is not exercising judgment. That claim needs no marking scheme. It is readable straight off the replay.
02Why my own tests could never have caught it
Reviews of this component were constant. The loudest was a campaign of 44 agents that graded the whole system world class on 2 of 10 dimensions, competitive on 8, behind on 0.
That was honest work and I want to be fair to it. It measured coverage: whether a component existed for each concern a serious system should have. Coverage was real. Every component it looked for was there.
But every grader in it was built by me, ran on my task mix, and used my definition of success. None of them asked whether the router still varied its output, because it had not occurred to me to ask.
Every internal check inherits its author's assumptions: the task mix, the definition of success, the simulation code, the blind spots. A benchmark you did not write is a different instrument, and it disagrees with you in ways your own tests structurally cannot.
03I got the first reading wrong, and published it
My first report on the replay said we passed, and put 0.7699 up as proof. I read that as the shipped policy holding its own against an instrument built by strangers.
The number was real. The verdict was wrong.
I had added 5 percent forced exploration to the simulation to stop the policy sticking, and then failed to account for it. So the run scored a policy that wanders, while the one that ships stands still. What that artefact records is the score of an exploring run. It was never a pass mark for the configuration in production.
Even the external exam got graded by the family on the first pass. Keep that.
04Root cause: a good rule with nothing to feed it
Why does a tuned policy collapse onto one option? The answer is short and it is not exotic.
The policy will not trust a model's score until it has seen at least 5 observations of that model, which is sensible, because acting on one data point is how you get fooled. But at task zero no model has 5 observations. So the policy falls back to a default.
Only the default now accumulates observations. So only the default can ever reach 5. Every other model stays permanently unrated and is scored as negative infinity forever. The first arm touched is the only arm that can ever be touched.
On its own the abstention rule is good. Combined with zero exploration it is a trap that closes on the first task. My confirming detail: the entire tuning grid produced byte-identical output at zero exploration, so every parameter I had argued with myself about for weeks could not possibly have mattered.
One caveat the artefact insists on. In the simulation the unrated fallback is hardcoded to the cheapest model, while the live system falls back to the mid tier. So the live version of this trap is more expensive and less bad than the modelled one.
05The fix, and what the refutation bought
The fix is exploration: force a fraction of tasks onto a model the policy would not have chosen, so the other options can accumulate observations at all. I armed it in the same push, with a minimum detectable effect calculation attached, so the next verdict arrives with a sample size rather than a feeling.
Retuning against the benchmark also settled an argument I had been having with myself. The pessimism setting I had fussed over for weeks was a red herring: efficiency 11.63x at q=0.10 against 11.24x at q=0.75, barely a difference. The cost term was the real lever. Every row of that sweep was measured with exploration on, so none of them describes the config that shipped.
At cost weight 0 the policy holds 90.0 percent of always-expensive quality at 76.1 percent of its cost. At 100 it holds 77.0 percent of the quality at 7.4 percent of the cost. At 1000, 75.7 percent at 6.5 percent, which is where the cost term stops paying.
I recommended that setting rather than adopting it. The live decision rule still carries no cost term, and saying so is the honest state of this.
06How to check whether your router still chooses
This one is cheap and you do not need a benchmark for the first step.
If your policy has an abstention rule, a minimum sample size, or a confidence threshold, it has this failure mode available to it. The only question is whether anything supplies the observations that let the threshold be crossed. Usually nothing does.
07What this buys, stated plainly
Written the day before the replay, my own audit said flatly that no external benchmark had ever been run against this system. This was the first. It caught a policy that had stopped choosing, and no second one has run since.
So the rule is now written down: every comparative claim about this system is self-assessed until a second instrument reports.
Would my own tests have caught this eventually? Thirty days of them did not, and they were the only tests looking. If your system has never taken an exam someone else wrote, you do not know its grade. You know its homework average.
08How every number here was derived
These commands read live state, so re-running them will give different numbers from the ones quoted here, usually larger. That is the point of printing the commands rather than only the conclusions.
# the absorbing proof, the 0.7699 first reading and the 5% artifact behind it, the retune table sed -n '80,102p' ~/.claude/AIOS-POST-V3-COMPLETION-2026-07-28.md # RouterBench: precomputed outcomes, 11 models, the first external measurement grep -n -B2 -A6 "RouterBench, and the refutation" ~/.claude/HANDOFF-aios-2026-07-28-30h-complete.md # the 44-agent campaign was self-run; no external benchmark had ever run before this grep -n -A4 "world-class 2/10" ~/.claude/AIOS-AUDIT-2026-07-26/VERDICT-v2.md # the abstention gate: minimum observations before a score is trusted grep -n "MIN_N\|min_n" ~/.sgnk/bin/routing-bandit-decide.sh # is exploration armed right now, and at what rate grep -n "EXPLORE" ~/.claude/settings.json ~/.sgnk/bin/routing-bandit-decide.sh # the cheapest check of all: how many distinct models did the policy pick jq -r '.model' ~/.sgnk/traces/*.jsonl | sort -u | wc -l