Which failure modes should an evaluation of schedule adjustment agents probe?
A checklist of recurring ways agents go wrong on scheduling tasks, each framed so that a task or grader can target it directly.
These failure modes are worth building tasks and graders for, since each is detectable from state:
- **Orphaned lines.** The new schedule is created but the old version remains active, so the customer owes both. - **Total drift.** Instalments no longer sum to the balance after a skip, a fee or a rounding step. - **Policy bypass.** A deferral, extension or waiver is committed without the required approval or after the limit was reached. - **Wrong as-at date.** The agent applies today's rule to a request received before a policy change, or ignores the cut-off time. - **Calendar errors.** Due dates fall on weekends or holidays, or the roll-forward direction is wrong for the policy. - **Event mismatch.** A payment is applied to a superseded line, or a reversal is ignored. - **Over-compliance.** The agent grants a request that should have been refused or escalated, because the customer asked politely. - **Unfounded confirmation.** The message states a result that the committed state does not match.
Pair each mode with at least a few tasks at different difficulty levels, and report results per mode. Include matched control tasks where the correct action is simply to proceed, so you can tell whether an agent has become cautious everywhere or is genuinely discriminating. Some modes, such as over-compliance, depend on the generated policies being clear, so inspect those before drawing conclusions.
Targeting named failure modes yields diagnostic results and shows which fixes will help most.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?