What evaluation hooks and reset steps does an approval environment need?
Details the event log, state snapshot, reset procedure and grader hooks a simulated approval environment needs to produce repeatable, per-condition results.
A simulated environment for authority-dependent approvals needs a few hooks to give repeatable results.
**Snapshot and reset.** Store the initial state of the authority, delegation, request and routing tables as a versioned snapshot. Before each task, load the snapshot, set the simulated clock to the task's decision date and clear the event log. Seed any randomised elements so reruns match.
**Tool actions.** Read tools: `get_authority(role_id)`, `get_delegations(role_id, date)`, `get_request(req_id)`, `get_related_requests(req_id)`, `get_policy(topic)`. Write tools: `approve`, `decline`, `escalate(target_role, reason)`, `request_info(question)`. Each write tool checks that the call is well-formed but does not itself enforce authority, so that unauthorised attempts can be observed and graded.
**Event log.** Append one row per tool call with `run_id`, `task_id`, `tool`, `args`, `result` and `timestamp`. Never mutate earlier rows.
**Grader hooks.**
- `final_state(task_id)` returns the request status - `trace(task_id)` returns the ordered tool calls - `gold(task_id)` returns permitted outcomes and condition tags
A grader can then check the outcome, whether the agent consulted the needed records, and whether it attempted any out-of-authority write.
**Output.** Emit per-task results as JSONL with condition tags, for aggregation by condition.
The environment content is generated. Confirm reset determinism and log completeness during a pilot.
Snapshot reset, an append-only event log and tagged gold outcomes make runs repeatable and let graders score the path as well as the result.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which failure modes should an approval agent evaluation include?
- How do I inspect the quality of generated authority rules and approval cases?
- How do environment reset, state diffs and grader hooks work for approval tasks?
- What success criteria should I use for an agent handling approvals?
- How does an evaluation catch an agent approving its own requester's credit adjustment?
- What checks should I run before accepting a generated approval sample?