What reset, state-diff and scoring hooks does an evaluation harness need?

An evaluation harness needs three things from the environment: a way to reset, a way to observe state, and a way to score.

**Reset.** Each case has a `seed` and a `start_state_ref`. A `reset(case_id)` call restores all tables to the starting snapshot, including tickets and holds, so repeated runs are comparable.

**Observation.** After each action the environment returns the tool result. A separate `get_state(table, key)` view can be exposed to the grader only. Keep grader-only views apart from the tools the model can call.

**Scoring hooks** - `on_step(case_id, step, action, state_hash)` for logging - `on_finish(case_id)` returns a state diff against `expected_state` - `score(case_id)` returns per-field results and a list of forbidden actions taken

Example score record (JSONL): `{"case_id":"CASE-0412","fields":{"status":"pass","schedule":"fail"},"forbidden":["release_drawdown"],"claimed_complete":true}`

**Practical notes** - Compare derived tables by value, with stated tolerance rules for rounding in schedules. - Log the model's final claim separately from the state. - Version the policy and grader together so results from different releases are not mixed. - Agree the adapter: a Python function set, an HTTP service or an MCP tool surface, decided in scoping.