How are graders, state diffs and resets wired into a schedule task harness?
A technical outline of how a harness can load seeded state, expose tools, log calls, diff final state against expectations and reset between runs, with per-step and outcome scoring.
A schedule task harness can be small. The pieces below are the ones most worth agreeing before a pilot.
**Lifecycle.**
- `reset(seed)` loads the snapshot for the task and sets the simulated clock. - The agent receives the request, the policy store and the tool list. - Each tool call is appended to an event log with arguments, result and timestamp. - `grade(task_id)` runs after the agent stops.
**Grader inputs.** `expected_final_state`, `allowed_outcomes` (act, escalate, request information, decline), `required_steps` with ordering constraints, and `forbidden_changes` such as edits to paid instalments.
**Outputs.** A JSON result per task: `outcome_class_correct`, `state_diff` listing fields that differ, `step_scores`, `order_violations`, `unauthorised_actions` and a total. Keeping the parts separate lets you chart failure modes.
**Determinism.** Fix the seed, the clock and tool latency. Where a tool returns generated text, store it with the snapshot so reruns are identical.
**Hooks.** A pre-grade hook can run your own checks, and a post-run hook can export the full trajectory to JSONL for later training.
The material is generated, so inspect a sample for graders that accept more than one valid answer, such as two correct ways to split a balance.
A harness that diffs final state, reads the tool-call log and resets deterministically gives repeatable scores you can compare across model versions.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?