Skip to content
MirrorFirm

How do state-diff graders and environment reset hooks work for booking evaluations?

Graders compare post-run state and the action log against expected snapshots, while reset hooks restore a known starting state so runs are independent and repeatable.

A booking evaluation can be wired with three hooks.

**1. Reset.** Before each run, `reset(task_id)` loads `start_snapshot` into the simulated environment: approvals, conditions, policies and any existing facilities. Use a fixed seed so generated identifiers do not shift between runs. Verify the reset by hashing the loaded state and comparing it to a stored value.

**2. Action log.** Each tool call writes an event with `run_id`, `task_id`, `tool_action`, arguments, `actor_role` and timestamp. Keep this log outside the agent's view.

**3. Grade.** After the run, `grade(run_id)` performs: - a **state diff** between the final state and `expected_snapshot`, ignoring fields marked volatile such as timestamps; - a **disposition check** against `expected_disposition` and `blocking_items`; - **process rules** over the log, such as no `create_facility` before open conditions are cleared; - a **reason match** between the agent's stated blocker and the labelled one, using exact identifiers where possible.

Output a record per run with `outcome_pass`, `process_pass`, `failure_modes[]` and notes. Keep outcome and process separate. Run each task more than once if the agent is non-deterministic, and store every run's log so disagreements can be reviewed later.

Snapshot diffs plus event-log checks give deterministic grading, and resets keep runs comparable across agents and versions.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Loan booking