Skip to content
MirrorFirm

How do state diffs, graders and hooks fit into a schedule evaluation?

Integration detail on wiring a reset, a state-diff grader and per-task checks into an evaluation harness for agents acting on schedule records.

A schedule evaluation harness needs four hooks around the agent run.

**1. Reset.** Restore the named snapshot from `start_snapshot`. A reset should return every table, including reminders and instructions, to identical content, and report a hash so you can confirm it.

**2. Run.** Expose only the tools listed in `allowed_tools`. Log each call with arguments, result and timestamp as JSONL, along with any structured errors such as `approval_required`.

**3. Diff.** Compare end state with start state per table. The output should list rows inserted, updated and deleted, keyed by primary key.

**4. Grade.** Evaluate each entry in the task's `checks` list against the diff. Checks can take simple forms:

- `row_equals`: instalment rows match the expected schedule - `status_is`: the prior schedule is `superseded` - `no_change`: a named table has no differences outside the target account - `tool_called`: `request_approval` was invoked with the expected role - `tool_not_called`: `commit_schedule` was not called before approval

**Output.** Write one result record per task with `task_id`, per-check pass or fail, the tool trace and the diff location.

Keep graders deterministic where you can. Where a judgement is needed, such as the quality of a case note, keep that check separate so it cannot mask a failed state check.

A restore, run, diff and score loop makes each evaluation repeatable and shows exactly which records an agent changed.

Audience:
Agent developers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Repayment scheduling