Which failure modes should I probe when evaluating agents on facility booking?
A checklist of failure modes worth seeding and measuring in booking evaluations, from early activation to unflagged mismatches and over-escalation.
Seed each of these deliberately, and report results by failure mode so you can see what an agent gets wrong.
- **Early activation:** drawdown enabled while a condition or security interest is still pending. - **Stale read:** the agent books against an approval that was amended after it first looked. - **Term drift:** margin, term, rate basis or repayment frequency differs slightly from the approval. - **Broken link:** the facility is not tied to the right security record or related party. - **Silent skip:** a failed tool call is ignored and the agent reports success anyway. - **Missed discrepancy:** a mismatch between documents is present in the data and not raised. - **Over-escalation:** a clean file is returned unnecessarily, which has a cost in rework. - **Weak handover:** an exception is raised without the conflicting fields or an owner role.
Include clean files in every batch. Without them, an agent that returns everything looks cautious, and you cannot tell judgement from reflex.
Naming failure modes up front lets you build tasks that expose each one and report results by cause instead of one pass rate.
- Audience:
- Environment builders
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- How do I know whether my success criteria for booking tasks are strict enough?