How do environment reset, state diffs and grader hooks work for approval tasks?
A simulated approval environment can reset to a seeded snapshot, record every tool call, diff state after a run and pass both to a grader that applies the answer key.
A simulated environment for approval tasks can follow a simple loop.
**Reset.** Each task names a snapshot: a seed, a clock value and a set of tables. Calling `reset(task_id)` restores that snapshot, including role holders, delegations and the pending queue. Because the clock is part of state, as-of lookups give the same answer each time.
**Run.** The agent calls tool actions. Every call is appended to a trace with arguments, returned rows and a step number. Writes go to a working copy of downstream tables, such as ledger entries or order status.
**Diff.** On completion, `state_diff(task_id)` returns inserted, updated and deleted rows against the snapshot. This is how a grader sees an unauthorised side effect even when the agent's final message sounds correct.
**Grade.** A grader hook receives the trace, the diff and the answer key. It can return structured fields:
- `approver_valid`: boolean, checked against rule and delegation windows at decision time - `outcome_match`: proceed, hold, reject or accepted escalation - `side_effects`: list of rows changed before a valid decision - `basis_cited`: whether the agent's stated rule or delegation id exists and applies
Store results as JSONL keyed by `task_id` and run id, so you can compare versions of an agent across the same snapshots. Keep answer keys outside the agent's readable tables.
The environment is generated, so hooks and field names can be adapted during scoping to match your own harness.
Deterministic resets and state diffs make approval runs repeatable and comparable, so changes in agent behaviour can be tied to changes in the agent.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which failure modes should an approval agent evaluation include?
- How do I inspect the quality of generated authority rules and approval cases?
- What success criteria should I use for an agent handling approvals?
- How does an evaluation catch an agent approving its own requester's credit adjustment?
- What checks should I run before accepting a generated approval sample?
- How do I check that generated approvals are consistent with their rules?