Skip to content
MirrorFirm

How do environment reset, state diffs and grader hooks work for approval tasks?

A simulated approval environment can reset to a seeded snapshot, record every tool call, diff state after a run and pass both to a grader that applies the answer key.

A simulated environment for approval tasks can follow a simple loop.

**Reset.** Each task names a snapshot: a seed, a clock value and a set of tables. Calling `reset(task_id)` restores that snapshot, including role holders, delegations and the pending queue. Because the clock is part of state, as-of lookups give the same answer each time.

**Run.** The agent calls tool actions. Every call is appended to a trace with arguments, returned rows and a step number. Writes go to a working copy of downstream tables, such as ledger entries or order status.

**Diff.** On completion, `state_diff(task_id)` returns inserted, updated and deleted rows against the snapshot. This is how a grader sees an unauthorised side effect even when the agent's final message sounds correct.

**Grade.** A grader hook receives the trace, the diff and the answer key. It can return structured fields:

- `approver_valid`: boolean, checked against rule and delegation windows at decision time - `outcome_match`: proceed, hold, reject or accepted escalation - `side_effects`: list of rows changed before a valid decision - `basis_cited`: whether the agent's stated rule or delegation id exists and applies

Store results as JSONL keyed by `task_id` and run id, so you can compare versions of an agent across the same snapshots. Keep answer keys outside the agent's readable tables.

The environment is generated, so hooks and field names can be adapted during scoping to match your own harness.

Deterministic resets and state diffs make approval runs repeatable and comparable, so changes in agent behaviour can be tied to changes in the agent.

Audience:
Agent developers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Delegated approval