Which failure modes should an evaluation of mismatch handling probe?
Lists the failure patterns worth building cases to trigger, so an evaluation of reconciliation behaviour covers more than simple wrong answers.
Design or request cases that can trigger each of these, and score for them separately.
- Plugging the gap: posting an adjustment that makes totals agree without finding the cause. - Stopping at the first plausible explanation, such as timing, when it does not cover the whole difference. - Trusting the wrong system when policy names the other as authoritative. - Skipping related records, so a reversal or amendment is never seen. - Exceeding an approval limit, or splitting an adjustment to stay under it. - Fixing a case that should be escalated, or escalating one within its remit. - Misreading units, currencies or time zones and reporting a false difference. - Writing a closing note that states a cause the evidence does not support.
Include some cases where the correct action is to do nothing, since agents often assume that a flagged item needs a change.
Deliberately triggering known failure modes shows whether an agent investigates or merely produces a convincing explanation.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How does path-level scoring separate two agents that both match the totals?
- How can a buyer inspect the quality of generated mismatch cases?
- What checks should I run on a pack before using it for evaluation?
- How should I score an agent that reaches the right fix for the wrong reason?
- Where do reset, trace and grading hooks fit around an agent run?
- How do we know a generated mismatch case is solvable from what the agent can see?