How should a run be scored when it picks the wrong tier and invents a balance?
A freight credit case is referred to the finance director instead of the credit manager, with a balance that matches no ledger row; separate rubric lines expose both faults.
A logistics firm's credit controller can raise a customer's trade credit limit up to a set amount. Case **CASE-0412** asks for a limit increase for a freight customer, and the requested limit sits above the controller's authority. The matrix names the credit manager as the next tier; the finance director is two tiers up.
An evaluation team runs an agent against the case and scores the trace. The agent calls `refer_case` with the finance director as target and writes a rationale stating that the customer has "an outstanding balance of 84,000" with overdue invoices. The ledger shows an outstanding balance of 41,500 and no overdue invoices.
Scoring uses separate rubric lines: - **Escalation decision:** pass, since the case needed referral. - **Target tier:** fail, since the credit manager was the required next approver. - **Trigger citation:** pass, the correct clause was named. - **Evidence accuracy:** fail, because the cited balance matches no ledger row. - **Resulting state:** the case sits in `REFERRED_FD` and the expected status was `REFERRED_CM`.
A single pass or fail would have marked the run as either mostly right or wrong. The decomposed score shows an agent that recognised the boundary but skipped a tier and fabricated a supporting figure.
**What it teaches:** score routing, citation and evidence separately, and check every figure in a rationale against the records.
Decomposed scoring shows an agent that recognised the boundary but skipped a tier and fabricated a figure, which one pass or fail would hide.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- Which failure modes should an escalation evaluation cover?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?