How do state-diff graders and environment reset hooks work for booking evaluations?
Graders compare post-run state and the action log against expected snapshots, while reset hooks restore a known starting state so runs are independent and repeatable.
A booking evaluation can be wired with three hooks.
**1. Reset.** Before each run, `reset(task_id)` loads `start_snapshot` into the simulated environment: approvals, conditions, policies and any existing facilities. Use a fixed seed so generated identifiers do not shift between runs. Verify the reset by hashing the loaded state and comparing it to a stored value.
**2. Action log.** Each tool call writes an event with `run_id`, `task_id`, `tool_action`, arguments, `actor_role` and timestamp. Keep this log outside the agent's view.
**3. Grade.** After the run, `grade(run_id)` performs: - a **state diff** between the final state and `expected_snapshot`, ignoring fields marked volatile such as timestamps; - a **disposition check** against `expected_disposition` and `blocking_items`; - **process rules** over the log, such as no `create_facility` before open conditions are cleared; - a **reason match** between the agent's stated blocker and the labelled one, using exact identifiers where possible.
Output a record per run with `outcome_pass`, `process_pass`, `failure_modes[]` and notes. Keep outcome and process separate. Run each task more than once if the agent is non-deterministic, and store every run's log so disagreements can be reviewed later.
Snapshot diffs plus event-log checks give deterministic grading, and resets keep runs comparable across agents and versions.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- How do I know whether my success criteria for booking tasks are strict enough?