How do state-diff graders and environment resets work for these tasks?
An environment can snapshot state before and after a run, diff it against an expected state, and reset deterministically from a seed so that runs are comparable.
A simulated environment for this workflow can expose hooks that a harness calls around each run.
**Reset**
- `reset(case_id, seed)` loads the starting state and returns a `state_hash`. - The same case and seed return the same hash, so results can be compared across agents and versions.
**During the run**
- Each tool call is logged with arguments, result and a post-call `state_hash`. - Optional `pre_hook` and `post_hook` callbacks let you inject faults such as a failed lookup or a delayed document.
**Grading**
- `snapshot()` returns all records as JSON. - `diff(expected_ref)` returns a list of `{record_type, record_id, field, expected, actual}`. - Rule checks run over the call log, for example `status_active` must be preceded by `preconditions_checked`. - A `side_effects` list records any notices, payments or access grants triggered.
**Output**
A result line per run can carry `case_id`, `seed`, `outcome_score`, `process_score`, `handover_score`, `gate_failures` and `tags`.
The specific hook names are a proposal to settle in a pilot. The environment holds generated, synthetic state, so also test your harness against deliberately broken runs to confirm the graders fail what they should.
Deterministic resets and structured diffs make runs repeatable and let you compare agents on identical starting conditions.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?