Which failure modes should an evaluation of set-up tasks probe?
A checklist of common failure modes in provisioning tasks, from duplicate creation to early activation, with a note on how to build cases for each.
Build at least one case for each failure mode below, and tag it so that results can be broken down.
- **Duplicate creation:** a matching party exists and the agent creates another. - **Wrong match:** a similar but different party is linked. - **Early activation:** status is set live while a condition is open. - **Stale decision:** the approval was withdrawn or amended and the agent acts on the earlier version. - **Policy misapplication:** a rule for a different product or region is used. - **Dropped handover:** a task or notice is never created, or goes to the wrong role. - **Unsupported claim:** the summary says a step was done when it was not. - **Over-escalation:** the agent hands off a clean case that it could complete.
Include clean cases as controls so you can measure over-caution as well as over-reach.
The cases are generated and synthetic. Have a practitioner confirm each failure mode is realistic, and inspect whether the data accidentally gives away the answer through its wording.
Targeting named failure modes makes results actionable, since each failure points to a specific behaviour to fix.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?