Any agent can be made to look good. Count the actions it completed, ignore the ones the user immediately undid, treat every time it declined as a failure of capability rather than an act of honesty, and you will produce a chart that climbs.
So before measuring anything I wrote down the ways this measurement could lie — as a numbered list, in a spec section titled “every one of these is a mistake already made here.” Not hypothetical failure modes. Ones already committed in this codebase.
The unit is an attempt, and refusal is a result
Every graded event resolves to one of eight outcomes. The two that matter:
refused-honestly— the system said it couldn’t, and that was true.ungradeable— we genuinely cannot tell — never a synonym for success.
An explicit rejected action with no successful retry is recorded as
refused-honestly, separate from the failure rate. The label is narrower than
“the agent said no”: it is reserved for a rejection the event rules can place,
and a successful retry supersedes it. The point is to stop an evidenced limit
from being scored as though the system tried and failed.
ungradeable carries a required reason. An unexplained one is the exact
silent failure the subsystem exists to prevent.
Four of the eight rules
An evidenced honest refusal is not a failure. Asks, rejected actions and errors stay separate, so an arm that asks more does not look like an arm that errors more.
An attributed reversal can make an action retroactively wrong. A committed
agent action becomes wrong when the user reverses it inside the open undo
window. A person undoing their own edit remains a correction and does not blame
the agent. Counting the first case as a clean commit is how a system reports
success at the moment its work is rejected.
Never derive an outcome from model self-report. No field may come from what
the model said it did — because an action here was once acknowledged
success: true while the document was byte-identical.
Sample size travels with every number. No rate is reported without its n,
and below a stated threshold a comparison returns underpowered by name,
never as “no effect”. Reporting no effect from four sessions is the most likely
way this apparatus would mislead its own author.
The doctrine as an assertion
The rules are only worth the tests that pin them. The keystone one: a session
consisting entirely of attempts already classified refused-honestly must
score 0% failure, not 100%.
Every derived rate also gets a direction test — construct a session where the
rate must go up, and assert that it does — because a correction rate here was
once silently inverted by widening its denominator, so an arm that provoked
more refusals scored as needing less correction.
There is no statistical inference: no p-values, no confidence intervals.
underpowered is a crude threshold on n, deliberately, because a crude
honest signal beats a sophisticated one nobody can audit.
What the humans found
Sittings join the same corpus, and the prose stays: a person’s judgement about why something failed is the most valuable field in the whole record and is not flattened into an enum.
The last frame is the one I would point at. A test was left pending because the fix for it had been verified on a neighbouring code path rather than driven directly — “that is inference”. It would have been trivially easy, and completely wrong, to mark it passed.
What this does not claim
This grades whether the system did what was asked. It does not grade whether what it produced was any good.
And the corpus is small: two documented human sittings, plus development sessions. There is no population here, no randomisation — which arm a sitting runs is a human decision — and no finding I would present as a result. The instrument is built and tested; it has not yet been pointed at enough people to say anything. That is the honest state, and by rule 8 it travels with every number this thing will ever report.