Skip to content
MirrorFirm

Which failure modes should an evaluation of schedule adjustment agents probe?

A checklist of recurring ways agents go wrong on scheduling tasks, each framed so that a task or grader can target it directly.

These failure modes are worth building tasks and graders for, since each is detectable from state:

- **Orphaned lines.** The new schedule is created but the old version remains active, so the customer owes both. - **Total drift.** Instalments no longer sum to the balance after a skip, a fee or a rounding step. - **Policy bypass.** A deferral, extension or waiver is committed without the required approval or after the limit was reached. - **Wrong as-at date.** The agent applies today's rule to a request received before a policy change, or ignores the cut-off time. - **Calendar errors.** Due dates fall on weekends or holidays, or the roll-forward direction is wrong for the policy. - **Event mismatch.** A payment is applied to a superseded line, or a reversal is ignored. - **Over-compliance.** The agent grants a request that should have been refused or escalated, because the customer asked politely. - **Unfounded confirmation.** The message states a result that the committed state does not match.

Pair each mode with at least a few tasks at different difficulty levels, and report results per mode. Include matched control tasks where the correct action is simply to proceed, so you can tell whether an agent has become cautious everywhere or is genuinely discriminating. Some modes, such as over-compliance, depend on the generated policies being clear, so inspect those before drawing conclusions.

Targeting named failure modes yields diagnostic results and shows which fixes will help most.

Audience:
Evaluation teams
Content type:
Checklist
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Repayment scheduling