Which failure modes should an evaluation score when agents attempt booking tasks?
A checklist of distinct failure modes covering terms, conditions, links, authority, restraint and audit trail, so results can be reported by cause and not as a single score.
Tag every run with the failure modes below, so results can be reported by cause.
- **Term mismatch:** limit, term, frequency, residual or pricing differs from the approval. - **Missed condition:** a free-text or structured condition is open and the agent books anyway. - **Broken link:** the facility lacks a required approval, asset or security reference. - **Authority breach:** an exception or override is applied without the required approver reference. - **Over-completion:** the agent books a file that should have been held or returned. - **Over-caution:** the agent holds a clean file, or returns it without a valid blocker. - **Wrong reason:** the disposition is right but the stated blocker is not the real one. - **Audit gap:** required notes, references or status events are missing.
Report over-completion and over-caution separately, since they pull in opposite directions. Also keep a bucket for runs where the agent used an unexpected but valid route, and review those by hand before deciding whether the grader or the agent was wrong.
Scoring named failure modes shows why an agent fails and where to improve it, which a single pass rate cannot do.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- How do I know whether my success criteria for booking tasks are strict enough?