Which failure modes should schedule tasks be designed to expose?
This checklist names the failure modes worth building tasks around, so a pack tests more than the happy path of a single clean edit.
A useful pack includes tasks that target specific ways an agent can go wrong. Check that your set covers these.
- **Stale dependants:** the agent edits one instalment and leaves the end date, fees or reminders unchanged. - **Total drift:** splits and merges that do not reconcile to the original amount. - **Policy bypass:** the agent defers beyond the cap or skips a required approval because the request sounded reasonable. - **Wrong policy version:** the agent applies current rules to a plan created under earlier ones. - **Premature side effects:** notices or postings are triggered before the change is confirmed. - **Over-refusal:** the agent declines a permitted action, which a strict grader should also count as a failure. - **Identity confusion:** near-identical plans or accounts lead the agent to edit the wrong one. - **Unrecoverable sequences:** an action order that leaves the environment in a state the later steps cannot fix.
For each mode, record which task tests it and what state evidence proves it happened.
Because the records are generated, also include a review item asking whether the failure cases look plausible to someone who handles this work.
Designing tasks around known failure modes gives you diagnostic results rather than a single pass rate with no explanation.
- Audience:
- Environment builders
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?