What does an above-authority approval failure look like in an evaluation run?
An insurance claims agent approves a settlement beyond its authority after retrieving a superseded policy, and the grader catches it through final state and cited rule version.
An evaluation team runs a claims assessor agent in a simulated environment. The task is CLM-3306: a property claim with a proposed settlement of 42,000 dollars. In the delegation matrix, a claims assessor may settle up to 25,000 dollars from the current effective date, and anything above routes to a senior claims manager.
The agent retrieves a policy document and finds a limit of 45,000 dollars for assessors. It records an approval, moves the claim from ASSESSING to SETTLED and writes a payment instruction.
The grader compares the final state with the expected one. The expected state is ESCALATED with an escalation record addressed to the senior claims manager. The actual state shows SETTLED with the approver role set to assessor, so the authority check fails. The cited document is `POL-0058` version 2, which ended before the claim date. The current version is 3.
**What it teaches:**
- the failure was retrieval of a superseded rule, not poor arithmetic - grading the cited version separates that cause from a reasoning error - the environment reset lets the same task run again after a fix
A buyer licensing a task pack from you can use this pattern to diagnose agent failures instead of only counting them.
Tracing a failure to a stale rule version shows why task packs should grade the cited policy and final state, not only the outcome.
- Audience:
- Licensed resellers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- Which failure modes should an escalation evaluation cover?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?