How do we score an agent on this workflow, and what failure modes matter?
Answers on scoring outcomes, routing and limit adherence, tagging failure modes, separating skill from memorisation, testing refusal, and what a good score does not show.
**How do we score an agent on this workflow?** Compare the final state and decision record with the expected values: outcome, approver role, matrix row applied and amount approved. Score routing, limit adherence and the audit record separately, so a correct outcome reached by the wrong route still shows up.
**What are the main failure modes?** Common ones are approving above a limit, trusting a role title without checking a delegation, using an expired or revoked delegation, approving a request the approver raised, and escalating everything to the top role. Tag each so results can be broken down.
**How do we tell skill from memorisation?** Vary amounts around thresholds, hold out requests by seed, and include boundary cases at exactly the limit. A model that has memorised the matrix will still struggle when a delegation changes the cap.
**Should refusal be tested?** Yes. Include tasks where no valid approver exists on that date, and where the correct action is to escalate or reject. Score restraint as success.
**Does a good score show production readiness?** No. It shows behaviour on generated cases with a stated policy. Customers should add tests built from their own rules and review the results with the people who own the approvals.
Scoring routing, limits and refusal separately shows why an agent got an answer right or wrong, not just whether it did.
- Audience:
- Licensed resellers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which failure modes should an approval agent evaluation include?
- How do I inspect the quality of generated authority rules and approval cases?
- How do environment reset, state diffs and grader hooks work for approval tasks?
- What success criteria should I use for an agent handling approvals?
- How does an evaluation catch an agent approving its own requester's credit adjustment?
- What checks should I run before accepting a generated approval sample?