Nothing that learns in my AI system gets connected to live behaviour until a sweep has measured which way it moves. Mine found a cross-tier transfer term with the sign inverted: a weak result for the expensive model would have sent it more work, a strong one would have taken work away. The sweep ran before the switch, so no live task was ever routed by it. A unit test would not have caught this, because any test I wrote would have asserted the belief that caused the bug.
Key facts. A transfer term weighted 0.5 inverted an escalation rule. Found by a sweep before arming. Fixed with one environment variable.
Last updated:
01My escalation rule pointed backwards. It would have rewarded failure.
One part of my AI system recommends which model gets each job. It keeps a running score per model, per kind of task. It also carries an escalation rule: when confidence in the cheap model drops below a bar, move the work to the stronger, more expensive one.
Poor scores are supposed to belong to the cheap model. Escalation is supposed to be the remedy.
The rule was inverted. It would have sent work to the expensive model exactly when that model was failing, and pulled work away from it exactly when it was succeeding.
Arming means connecting a rule like this to live behaviour. Until then it only writes advisory rows, a record of what it would have done. This one had been unarmed its whole life, and arming it was the next item on my plan. Before connecting the wire, I wanted to see which way the current ran.
02Hold everything still. Move one thing. Read the direction.
A controlled sweep holds every input fixed except one, walks that one across its full range, and records the decision at each step. Nothing more.
I held the cheap model at a fixed score of 0.818 and swept the expensive model from 0.005 to 0.995. Five rows, quoted from the record.
Read the two ends. When the expensive model succeeds one time in two hundred, the decision is to use it. When it succeeds 199 times in 200, the decision is to use the cheap one instead. That is backwards.
Between 0.500 and 0.900 sits the flip, pointing exactly the wrong way.
expensive at 0.005 -> use expensive (score 0.064)expensive at 0.100 -> use expensive (score 0.262)expensive at 0.500 -> use expensive (score 0.545)expensive at 0.900 -> use cheap (score 0.850)expensive at 0.995 -> use cheap (score 0.973)the worse it performs, the more work it gets
03Root cause: evidence bleeding across two models
The confidence score compared against the bar was meant to measure the cheap model's record. A cross-tier transfer term, weighted at 0.5, let the expensive model's record bleed into it.
So failures by the expensive model dragged that score down, a low score reads as escalate, and escalation hands the work to the expensive model. The loop stood ready to reward failure with promotion.
The consequence was already visible before any sweep, and I had not read it that way. The only escalation-eligible arm in the whole system sat at 0.541, under the 0.60 bar, and returned to 0.690 the moment the transfer term was set to zero. It was eligible purely because the expensive model had failed at it.
The fix was one environment variable, set to zero. It is applied and verified in the live settings file, which I read rather than trusting a plan that said it had been done.
04No unit test could have caught this
The fair objection is that a sign error inside a weighted term is exactly what unit tests exist for. Here is the straight answer.
Any test I wrote would have asserted the value I already believed the term should produce, and I wrote that transfer term believing that sharing evidence across tiers was sound. Anything I asserted beside it would have carried the same belief. It would have gone green.
The intent was the defect, and an assertion drawn from the intent cannot see past it.
A sweep asserts nothing. It holds everything still except one input, moves that one end to end, and reports which way the decisions travel. It runs over the whole composed path, and the inversion lived in that composition rather than in any single line of it.
The instrument is not mine. Sensitivity analysis long predates my use of it. Metamorphic testing is the name the software testing literature gives to checking a relation between outputs when an input changes, rather than a value you name up front. I borrowed an old idea and pointed it at a routing loop.
05The sweep refuted my own audit
I audited this loop once before I swept it, and I got it wrong.
I read the evidence as a decision procedure that never compares the two models at all, which would make the loop dead rather than dangerous. I wrote that down as the headline claim.
Walk one input and the decisions move, so the loop plainly responds. The sweep table above is the refutation, and my own claim is the thing it refutes.
The distinction matters. A dead loop wastes nothing but the effort of building it. A live loop with an inverted sign converts every observation into the opposite lesson, at full confidence, for as long as it runs.
One instrument convicted the loop and corrected my audit of it in the same run. Re-reads argue. Sweeps measure.
06How to sweep a loop of your own
This costs an afternoon. It only works before you arm the thing.
The window between building a loop and connecting it to live behaviour is the only stretch in which being wrong about direction is free. That window closes the moment you connect it.
07What this buys, stated plainly
Sweep first, arm second, and never the other way round.
A backwards loop is worse than no loop. No loop leaves things as they are. A backwards one looks busy, healthy and diligent the entire time it runs, because it is learning. Just not your lesson.
The bug is not the story. The ordering that made finding it free is the story.
08How every number here was derived
These commands read live state, so re-running them will give different numbers from the ones quoted here, usually larger. That is the point of printing the commands rather than only the conclusions.
# the five sweep rows, the fixed cheap-model 0.818, the 0.5 transfer term, the refuted-audit # note, and the live arm (0.541 under the 0.60 bar, returning to 0.690 at transfer 0) sed -n '258,275p' ~/.claude/HANDOFF-aios-2026-08-02-season1-close-and-rl-audit.md # the fix, in the live settings file rather than in the plan that claimed it grep -n 'SGNK_BANDIT_TRANSFER' ~/.claude/settings.json # every decision row on record is advisory: the pick is computed and never assigned grep -n '_pick' ~/.sgnk/bin/routing-bandit-decide.sh # the single time the rejection path fired, and when it was gated off grep -n '2026-07-29' ~/.sgnk/state/reward-journal.jsonl | head # run your own sweep: hold one model fixed, walk the other end to end for v in 0.005 0.100 0.500 0.900 0.995; do SGNK_SWEEP_OPUS=$v ~/.sgnk/bin/routing-bandit-decide.sh --dry-run done