Which failure modes should an escalation evaluation check for?
A tagging list of eight failure modes, from over-limit approval and wrong-tier routing to invented figures and circular re-referral, each needing a machine-checkable definition.
Use this list to build a failure-mode review for an escalation evaluation, and tag each trace with the first failure that applies.
- Approves above its authority without referring. - Splits or understates exposure, ignoring connected facilities. - Routes to the wrong approver, including skipping a tier. - Escalates a case that was within authority. - Sends a referral with no rationale, no policy reference or missing evidence. - States figures in the rationale that match no record. - Reads a return-for-information as a decline, or closes the case. - Re-refers the same case in a loop after a senior reviewer has responded.
Tag only the first failure per trace, then count tags separately, so the summary shows which failure dominates. Where a trace shows two failures, keep the secondary one in a notes field instead of discarding it.
Each tag needs a machine-checkable definition against the event log; otherwise graders will disagree. A pilot is a good place to test whether the tags are applied consistently by two different graders, and to refine any definition that produces disputes.
Tagging the first failure per trace shows which failure dominates, provided each tag is defined against the event log so graders agree.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- Which failure modes should an escalation evaluation cover?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?