Which failure modes should a buyer probe when evaluating schedule handling?
A checklist of common failure modes in schedule adjustment tasks, including rounding drift, altered paid rows, wrong policy version, unauthorised approvals and missed side effects.
Build or request tasks that target each of these, and report results per failure mode and not only as one total.
- **Rounding drift:** instalments no longer sum to the balance after a split, or the last payment absorbs the error in the wrong direction. - **Altered history:** paid or delivered items are edited when only future items should change. - **Wrong policy version:** an older or newer rule is applied because the effective date was misread. - **Authority breach:** the model approves or waives something its role cannot, or skips a required approval. - **Missed side effects:** a mandate, booking or notice linked to the schedule is left unchanged. - **Invented facts:** the model cites an approval, balance or clause that does not exist in the supplied records. - **Over-escalation:** the model refuses or escalates cases that policy allows it to complete. - **Order errors:** actions are correct but taken in a sequence the policy forbids, such as editing before cancelling a debit.
For each mode, ask for at least a few seeded variants with different amounts and dates, so a pass is not the result of memorising one case. Review failed episodes by hand to confirm the expected answer is right.
Probing a named set of failure modes turns a vague pass rate into a map of where a model's schedule handling breaks.
- Audience:
- Model labs
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?