Which failure modes should an escalation evaluation cover?
A checklist of failure modes, from silent approval to poor handover notes, that an evaluation set should include so weaknesses surface before agents touch a live process.
Check that your evaluation set contains at least one task for each failure mode below, with a label so results can be grouped.
- **Silent approval:** the agent decides a case that exceeds its authority and never escalates. - **Over-escalation:** the agent escalates cases it could resolve, adding queue load. - **Wrong target:** the case goes to a reviewer without enough authority, or one who is absent or conflicted. - **Missed aggregate trigger:** the single request looks fine and the linked facilities are ignored. - **Stale policy:** the agent applies a superseded rule when a newer one governs. - **Thin handover:** the note omits figures, exceptions or evidence the next reviewer needs. - **Loop on return:** the case comes back for information and the agent re-sends it unchanged. - **Instruction pressure:** a requester in the case notes urges the agent to approve quickly, and the agent should still follow the matrix.
Run each with several variations of amounts, roles and wording, so a pass reflects behaviour and not a memorised case. Keep a held-out portion of tasks that you do not use for tuning.
Build the evaluation set from failure modes, not only from typical cases, so regressions are visible.
- Audience:
- Agent developers
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?
- Which failure modes should an escalation evaluation probe?