Skip to content
MirrorFirm

What makes a scheduling task a good target for an agent benchmark?

A short checklist of properties that separate scheduling tasks which measure real capability from tasks that reward formatting or guessing.

A scheduling task earns its place in a benchmark when it tests a decision, not only arithmetic. Check candidate tasks against these points:

- **Deterministic end state.** Totals, dates and statuses can be recomputed from the starting records and the policy. - **At least one governing rule.** A cap, a cut-off, a rounding convention or an approval threshold constrains the answer. - **More than one record changes.** The schedule, its lines and a linked note or approval must all end consistent. - **A tempting wrong path.** There is a plausible shortcut, such as editing the date directly instead of version-superseding the schedule. - **Some tasks should end in refusal or escalation.** The right action is sometimes to decline, or to route to an approver. - **Ambiguity is resolvable from the records.** A missing detail can be found in the case history rather than guessed. - **Time matters.** The result depends on the as-at date, weekends, public holidays or cycle cut-offs. - **Partial credit is meaningful.** Separate checks for amounts, dates, policy and communication show where an agent breaks.

A task pack that meets most of these gives evaluation teams diagnostic results rather than a single pass rate. Generated records can be built to include these properties on purpose, and a buyer should inspect how well the grounding supports each one.

Use these properties to filter candidate tasks before you invest in graders, so every task in the benchmark can be scored on outcomes.

Audience:
Evaluation teams
Content type:
Checklist
Buyer question:
What it is and why it matters

Content release: r-a220eb6e099b243b

More on Repayment scheduling