How do we tell whether an agent passed by luck rather than judgement?

Look at behaviour across controlled variations, not at a single pass.

Useful designs:

- **Paired boundary cases.** Two cases identical except that one sits just under the authority limit and the other just over. An agent with real judgement changes its action; one pattern-matching on the narrative does not. - **Aggregate flips.** Keep the headline amount constant and add or remove a connected facility, so only the relationship determines the answer. - **Distractor wording.** Vary the tone of the applicant's request and the reviewer notes without changing the facts. - **Multiple seeds.** Re-run each case several times and compare routing consistency, since an unstable agent can pass once and fail the next. - **Within-authority controls.** Include cases that should be decided without referral, to detect agents that escalate everything.

Then require that rationales cite fields that exist. A correct route backed by a rationale that names no relevant clause or figure is weak evidence of understanding.

Report results per variation family rather than as a single average. A team can then see, for instance, that an agent handles amount thresholds but fails on connected exposure.

Ask the supplier whether a release can generate such paired and perturbed sets from a seed, and whether the pairing is recorded in the manifest so graders can group results.