The Brilliant Employee
The Brilliant Employee · 08

My Task Log Records What the AI Spent. Two Extra Fields Record Whether It Was Worth It

Cost and latency price a task. They cannot say whether the money bought anything. Two fields can, and no instrumentation standard carries the second one.

AuthorSagnik Mitra
Published2026-08-04
Reading timeabout 4 min
Deck11 slides
Download PDF

What is inside

  1. A cost log tells you what you spent and nothing about whether it was worth it
  2. What the standard covers, and what it cannot
  3. One real line, from 3 August
  4. One disagreement outweighs ten agreements
  5. I miscounted my own file, and here is how
  6. What one line buys, stated plainly
  7. How every number here was derived

Read a month of a normal AI task log and you know exactly what your system cost and nothing about whether the money bought anything good. Two fields fix that: what the model router recommended against what actually ran, and whether the human kept the work. OpenTelemetry's conventions already standardise the cost half. Neither they nor any vendor export carries a field for acceptance, because it is not a machine's to emit. Both verdicts have to land on the same line, joined by one id, in a file the worker does not write.

Key facts. One append-only line per finished task since 27 June, up to 33 fields. Every row pairs the machine verdict with the human one.

Last updated:

01A cost log tells you what you spent and nothing about whether it was worth it

Most of what a task log holds is bookkeeping. Tokens, milliseconds, a model name, a price. All of it real, all of it useful, and none of it able to answer the only question that matters once the invoice arrives.

Was the work any good?

A cost row cannot say. It records what was spent, and spending is not evidence of value in either direction. A cheap task can produce garbage. An expensive one can produce exactly what you needed.

Two fields on my rows answer it. What the router put forward against what actually ran, and whether I kept the work once I had read it.

You can rent the model. You cannot rent the part that tells you it was wrong.
what a normal cost log knows
tokens in and out
yes
latency
yes
which model ran
yes
was it any good
no
what two more fields add
what was recommended
yes
whether it was followed
yes
whether a human kept it
yes
was it any good
measurable
The left column prices a task. Only the right one can tell you if paying was worth it.

02What the standard covers, and what it cannot

The cost half of this is not mine to claim. OpenTelemetry's conventions for AI systems already settled the field names for model, tokens and latency, and my rules map onto them deliberately rather than inventing a private vocabulary.

What those conventions do not carry is a field for whether a human kept the work, and they could not. Acceptance is not a machine's to emit, so no instrumentation standard supplies it and no vendor export contains it.

That gap is what this fills. Vendors instrument what the agent spent. This instruments whether it was any good, and the two are useless apart.

The house rule says every task emits one line carrying an id and an acceptance field. Those two are named because they are load-bearing. Without the id, nothing joins. Without acceptance, nothing has an outcome. Every task, including the embarrassing ones, which are the point.

one row, by what each field is for
correlation_id
joins this row to everything else
model, tokens, latency
the cost half, standard names
decided_model
what the router recommended
assertion_pass
the machine verdict
accepted
the human verdict
The last field is the one no standard can supply and no machine can write.

03One real line, from 3 August

Here is a row from the small hours, redacted only where it names the session. A generation task at 00:12 UTC, sized to the cheap tier, run as a single shot.

Three keys carry the whole argument. The full row carries 33.

The machine check passed this task cleanly. One assertion of one, source machine-gate. Three keys later, acceptance says false.

The verifier said yes. I said no. Both verdicts sit on the same line, joined forever by the same id.

~/.sgnk/traces/2026-08-03.jsonl, one row, abridged
correlation_id   SESSION:16
gate_tier        floor
decided_model    sonnet
decision_mode    advisory
decision_real_n  0
model            claude-opus-5
input_tokens     838575  (838573 from cache)
output_tokens    9896
latency_ms       191000
assertion_pass   true
accepted         false
The last two lines disagree. That disagreement is the most valuable thing on the row.

04One disagreement outweighs ten agreements

Ten rows where the machine grader and I agree teach almost nothing. They are consistent with a well-calibrated grader and equally consistent with a grader that passes everything.

One row where the machine passes work I reject is a direct measurement of how miscalibrated the grader is. One row is not a calibration. It is the unit calibration is made of, and you can only collect it if both verdicts are written to the same place at the same time.

The second pair on that row says something else uncomfortable. The router recommended the cheap model. The model field says the expensive one ran anyway, because the decision mode is advisory: the pick gets logged, not enforced. And the observation count says zero, meaning the recommendation had no evidence behind it.

So the line confesses, on the system's behalf, that the learning loop is not closed. The routing component writes its homework down and does not yet read it.

machine verdict
assertions
1 of 1 passed
source
machine-gate
says
the work is fine
human verdict
accepted
false
source
me, after reading it
says
the work is not fine
Neither verdict alone is useful. The pair is a calibration point.

05I miscounted my own file, and here is how

Worth admitting, because it is exactly the kind of error the file exists to catch.

Counting the log directory the obvious way returns more files than there are days. I read the extras as evidence that the log had been running before the rule that mandated it, which would have been a pleasing story about the habit predating the policy.

Wrong. The glob printed in the commands below does not recurse, while find does. So what find picks up are step-level files sitting in a subdirectory, which are not daily files at all.

What the artefact actually says is narrower and duller than my reading of it. One file per day on which the log was written. 39 of them as of 4 August, with no missing dates since 27 June.

1Write one line per finished task, appended, never rewritten.
2Have the harness write it, not the worker. A worker that grades its own homework will also tidy its own transcript.
3Carry an id on every row, so a human verdict recorded later can be joined back.
4Record acceptance separately from any machine verdict, and keep both.
Step 2 is why the file is trustworthy at all.

06What one line buys, stated plainly

Every loop worth having needs those two fields and nothing fancier. Calibrating the grader needs machine grades joined to human verdicts. Routing by track record needs a recommendation and an outcome side by side, often enough to become a record. Drift detection needs dated rows. Nothing else.

It costs one line per task, which makes it the cheapest component in the system and the most binding. Everything I am allowed to claim in public is bounded by what this file can prove.

Right now it proves the system remembers everything and has learned a little. There is a written rule that the phrase self-improving is banned here until somebody proves that file is being read, and the ban has not been lifted.

1Cost data prices a task. It cannot tell you whether paying was worth it.
2Acceptance is not a machine field, so no standard and no vendor export will supply it.
3Both verdicts must land on the same row, joined by one id, or neither is usable.
4The harness writes the row. The worker never gets a handle on the file.
One line per task, and it is the most binding constraint in the system.
You cannot show a loop improved without rows from before it changed.

07How every number here was derived

These commands read live state, so re-running them will give different numbers from the ones quoted here, usually larger. That is the point of printing the commands rather than only the conclusions.

# the log: one dated file per day since 2026-06-27, no gaps
ls ~/.sgnk/traces/*.jsonl | wc -l
# -> 39. The glob does not recurse; find returns 41 because
#    traces/steps/ holds two step-level files. One file per
#    day on which the log was written.

# the dissected row: machine pass and human reject on the same line
jq -c 'select(.assertion_pass==true and .accepted==false)' \
  ~/.sgnk/traces/2026-08-03.jsonl | head -1

# how many fields a row actually carries
jq -r 'keys | length' ~/.sgnk/traces/2026-08-03.jsonl | sort -n | tail -1

# the calibration points: every row where the two verdicts disagree
jq -c 'select(.assertion_pass != null and .accepted != null and
              (.assertion_pass != .accepted))' ~/.sgnk/traces/*.jsonl | wc -l

# what the router recommended against what actually ran
jq -r 'select(.decided_model) | "\(.decided_model) -> \(.model)"' \
  ~/.sgnk/traces/*.jsonl | sort | uniq -c | sort -rn
All episodes