How does reviewing failed tasks reveal a boundary-amount failure mode?
An evaluation lead and reviewer trace a loan instalment failure to an agent that ignored a tolerance figure, then add rule-based slicing to the reporting.
The evaluation lead runs an agent across a task pack of loan repayment allocations set in Aster Financial's lending book. The reviewer opens the results table and filters for tasks where the final ledger differs from the reference.
Task T-0318 stands out. Bank line BANK-6120 pays an amount slightly under the instalment due on LOAN-4471. Policy says a shortfall within the tolerance is applied and flagged, while a shortfall outside it is part-applied with a case raised. The agent applied the amount and closed the instalment without a flag.
The reviewer checks the tolerance value in the policy record, then the amount in the bank line. The shortfall sits just outside tolerance. The agent had treated any small shortfall as acceptable and never read the policy figure.
The reviewer groups other failures by the same pattern and finds several tasks with boundary amounts that share it. The same agent handled clearly large and clearly tiny shortfalls correctly. The lead adds a boundary-amount slice to the reporting and records the failure mode as threshold blindness.
The consequence of the original error would be an instalment marked paid when it was not, with arrears interest not charged and a hardship review missed.
**What it teaches:** aggregate scores hide boundary behaviour. Ask for tasks tagged by the rule they exercise, so you can slice results by rule and amount band and find where an agent stops reading the policy.
Tag tasks by the rule they exercise, so aggregate scores can be sliced to expose behaviour at policy boundaries.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do I judge whether generated remittance and allocation records are realistic enough?
- Which failure modes should I test for in a payment matching agent?
- How would an evaluation score an agent on a hospital insurer remittance?
- What reset, logging and grading hooks should an evaluation environment expose?
- How should scoring handle cases where several allocations are defensible?
- How do I check generated data for grounding and disclosure risk?