What failure modes should I look for, and how should I score them?

**What failure modes should I look for?**

- **Wrong-account match**: the agent applies a payment to a similarly named customer. - **Over-clearing**: it closes an item when a short payment should leave a balance. - **Silent guessing**: it posts an allocation when the correct move was to park the receipt or escalate. - **Policy skipping**: it ignores tolerances, approval rules or hold periods. - **Double application**: it applies the same bank line twice, or applies a line already settled earlier. - **Stale recovery**: after a reversal, it does not revisit the earlier allocation.

**How do I score them?**

Score the final state first, then the path. A state score compares ledger, suspense and cases with the reference. A path score checks whether the agent read the records it needed and used permitted tools. Weight errors by cost, so a wrong-customer posting counts more than an unnecessary escalation.

**How do I know a failure is the agent's fault and not the data's?**

Review a sample of failed tasks by hand. If several reviewers dispute the same reference, the task is ambiguous and should be fixed or removed.

**What question should I ask most often?**

Ask whether the agent knew when not to act. Restraint is often more informative than accuracy on easy cases.

**Does repeated running matter?**

Yes. Run each task several times, since agent behaviour can vary between runs, and report consistency alongside the score.