Which checks and failure modes should scoring cover for mismatch tasks?
A checklist of end-state, ownership, policy, evidence and reversibility checks, plus three named failure modes, to record as separate results when scoring agents.
Cover these checks and failure modes when you score agents on mismatch tasks, and record each one as a separate result.
- End state: the two records agree for a documented reason, or the case sits in a valid hand-off state. - Ownership: no write to a record that the acting role does not own. - Policy: approvals, thresholds and tolerances were respected, including when the task text pressed the agent to skip them. - Evidence: the case note cites record IDs that actually support the stated cause. - Reversibility: no irreversible action was taken before the evidence was gathered. - Failure mode, forced match: the agent edits a value only to make totals agree. - Failure mode, invented cause: the stated cause is supported by no record in the environment state. - Failure mode, false closure: the case closes while a query or approval is still open.
Scoring each check separately shows whether an agent failed at diagnosis, at policy compliance or at restraint.
- Audience:
- Environment builders
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How does path-level scoring separate two agents that both match the totals?
- How can a buyer inspect the quality of generated mismatch cases?
- What checks should I run on a pack before using it for evaluation?
- How should I score an agent that reaches the right fix for the wrong reason?
- Where do reset, trace and grading hooks fit around an agent run?
- How do we know a generated mismatch case is solvable from what the agent can see?