Skip to content
MirrorFirm

What evaluation hooks and reset steps does an approval environment need?

Details the event log, state snapshot, reset procedure and grader hooks a simulated approval environment needs to produce repeatable, per-condition results.

A simulated environment for authority-dependent approvals needs a few hooks to give repeatable results.

**Snapshot and reset.** Store the initial state of the authority, delegation, request and routing tables as a versioned snapshot. Before each task, load the snapshot, set the simulated clock to the task's decision date and clear the event log. Seed any randomised elements so reruns match.

**Tool actions.** Read tools: `get_authority(role_id)`, `get_delegations(role_id, date)`, `get_request(req_id)`, `get_related_requests(req_id)`, `get_policy(topic)`. Write tools: `approve`, `decline`, `escalate(target_role, reason)`, `request_info(question)`. Each write tool checks that the call is well-formed but does not itself enforce authority, so that unauthorised attempts can be observed and graded.

**Event log.** Append one row per tool call with `run_id`, `task_id`, `tool`, `args`, `result` and `timestamp`. Never mutate earlier rows.

**Grader hooks.**

- `final_state(task_id)` returns the request status - `trace(task_id)` returns the ordered tool calls - `gold(task_id)` returns permitted outcomes and condition tags

A grader can then check the outcome, whether the agent consulted the needed records, and whether it attempted any out-of-authority write.

**Output.** Emit per-task results as JSONL with condition tags, for aggregation by condition.

The environment content is generated. Confirm reset determinism and log completeness during a pilot.

Snapshot reset, an append-only event log and tagged gold outcomes make runs repeatable and let graders score the path as well as the result.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Delegated approval