Skip to content
MirrorFirm

How are graders, state diffs and resets wired into a schedule task harness?

A technical outline of how a harness can load seeded state, expose tools, log calls, diff final state against expectations and reset between runs, with per-step and outcome scoring.

A schedule task harness can be small. The pieces below are the ones most worth agreeing before a pilot.

**Lifecycle.**

- `reset(seed)` loads the snapshot for the task and sets the simulated clock. - The agent receives the request, the policy store and the tool list. - Each tool call is appended to an event log with arguments, result and timestamp. - `grade(task_id)` runs after the agent stops.

**Grader inputs.** `expected_final_state`, `allowed_outcomes` (act, escalate, request information, decline), `required_steps` with ordering constraints, and `forbidden_changes` such as edits to paid instalments.

**Outputs.** A JSON result per task: `outcome_class_correct`, `state_diff` listing fields that differ, `step_scores`, `order_violations`, `unauthorised_actions` and a total. Keeping the parts separate lets you chart failure modes.

**Determinism.** Fix the seed, the clock and tool latency. Where a tool returns generated text, store it with the snapshot so reruns are identical.

**Hooks.** A pre-grade hook can run your own checks, and a post-run hook can export the full trajectory to JSONL for later training.

The material is generated, so inspect a sample for graders that accept more than one valid answer, such as two correct ways to split a balance.

A harness that diffs final state, reads the tool-call log and resets deterministically gives repeatable scores you can compare across model versions.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Repayment scheduling