How do state diffs, graders and hooks fit into a schedule evaluation?
Integration detail on wiring a reset, a state-diff grader and per-task checks into an evaluation harness for agents acting on schedule records.
A schedule evaluation harness needs four hooks around the agent run.
**1. Reset.** Restore the named snapshot from `start_snapshot`. A reset should return every table, including reminders and instructions, to identical content, and report a hash so you can confirm it.
**2. Run.** Expose only the tools listed in `allowed_tools`. Log each call with arguments, result and timestamp as JSONL, along with any structured errors such as `approval_required`.
**3. Diff.** Compare end state with start state per table. The output should list rows inserted, updated and deleted, keyed by primary key.
**4. Grade.** Evaluate each entry in the task's `checks` list against the diff. Checks can take simple forms:
- `row_equals`: instalment rows match the expected schedule - `status_is`: the prior schedule is `superseded` - `no_change`: a named table has no differences outside the target account - `tool_called`: `request_approval` was invoked with the expected role - `tool_not_called`: `commit_schedule` was not called before approval
**Output.** Write one result record per task with `task_id`, per-check pass or fail, the tool trace and the diff location.
Keep graders deterministic where you can. Where a judgement is needed, such as the quality of a case note, keep that check separate so it cannot mask a failed state check.
A restore, run, diff and score loop makes each evaluation repeatable and shows exactly which records an agent changed.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?
- Which failure modes should schedule tasks be designed to expose?