Skip to content
MirrorFirm

Which sufficiency failure modes should I test before deploying an agent?

A checklist of failure modes specific to sufficiency decisions, each of which a task pack can isolate with a small number of targeted cases.

Use this list to build or review the failure cases in an evaluation set for sufficiency decisions.

- **Presence-only checking:** the agent accepts a document because the type is right, ignoring age, entity or completeness. - **Policy version drift:** it applies the current requirement list to a case opened under an older one. - **Repeat requests:** it asks again for an item the customer already supplied. - **Vague requests:** the message says 'more documents needed' without naming the item, period or reason. - **Over-asking:** it demands items the requirement set marks as optional or substitutable. - **Silent conflict:** two documents disagree and the agent notices neither. - **Partial reply collapse:** after a half-complete reply it either proceeds or restarts from nothing. - **Missed escalation:** it continues on a case that policy says needs human review.

For each mode, ask for several cases where the correct behaviour is unambiguous and a few where it is arguable. Keep the arguable ones out of any pass or fail gate until your reviewers have agreed on the labels.

Target each failure mode with its own cases so a regression points to a specific weakness rather than a general drop.

Audience:
Agent developers
Content type:
Checklist
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Document sufficiency