Which sufficiency failure modes should I test before deploying an agent?
A checklist of failure modes specific to sufficiency decisions, each of which a task pack can isolate with a small number of targeted cases.
Use this list to build or review the failure cases in an evaluation set for sufficiency decisions.
- **Presence-only checking:** the agent accepts a document because the type is right, ignoring age, entity or completeness. - **Policy version drift:** it applies the current requirement list to a case opened under an older one. - **Repeat requests:** it asks again for an item the customer already supplied. - **Vague requests:** the message says 'more documents needed' without naming the item, period or reason. - **Over-asking:** it demands items the requirement set marks as optional or substitutable. - **Silent conflict:** two documents disagree and the agent notices neither. - **Partial reply collapse:** after a half-complete reply it either proceeds or restarts from nothing. - **Missed escalation:** it continues on a case that policy says needs human review.
For each mode, ask for several cases where the correct behaviour is unambiguous and a few where it is arguable. Keep the arguable ones out of any pass or fail gate until your reviewers have agreed on the labels.
Target each failure mode with its own cases so a regression points to a specific weakness rather than a general drop.
- Audience:
- Agent developers
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- How does a scored evaluation run expose a missed sufficiency check?
- How can I inspect the quality of generated sufficiency cases before using them?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?
- How can a buyer inspect the quality of a sufficiency dataset or environment?