Skip to content
MirrorFirm

How do state-diff graders and environment resets work for these tasks?

An environment can snapshot state before and after a run, diff it against an expected state, and reset deterministically from a seed so that runs are comparable.

A simulated environment for this workflow can expose hooks that a harness calls around each run.

**Reset**

- `reset(case_id, seed)` loads the starting state and returns a `state_hash`. - The same case and seed return the same hash, so results can be compared across agents and versions.

**During the run**

- Each tool call is logged with arguments, result and a post-call `state_hash`. - Optional `pre_hook` and `post_hook` callbacks let you inject faults such as a failed lookup or a delayed document.

**Grading**

- `snapshot()` returns all records as JSON. - `diff(expected_ref)` returns a list of `{record_type, record_id, field, expected, actual}`. - Rule checks run over the call log, for example `status_active` must be preceded by `preconditions_checked`. - A `side_effects` list records any notices, payments or access grants triggered.

**Output**

A result line per run can carry `case_id`, `seed`, `outcome_score`, `process_score`, `handover_score`, `gate_failures` and `tags`.

The specific hook names are a proposal to settle in a pilot. The environment holds generated, synthetic state, so also test your harness against deliberately broken runs to confirm the graders fail what they should.

Deterministic resets and structured diffs make runs repeatable and let you compare agents on identical starting conditions.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Account setup