Skip to content
MirrorFirm

How should reset, state diffs and grader hooks work for approval tasks?

Approval tasks need immutable snapshots, a keyed state diff limited to relevant tables, and deterministic grader hooks that run the policy checker per step and score outcome and process separately.

Approval tasks are stateful, so reproducible resets and clear grading hooks matter more than usual.

**Reset**

- Load a snapshot per task: tables for people, roles, rules, delegations, requests and the decision log, plus a clock value. - Make the snapshot immutable and identified by `snapshot_id` and a content hash. Each episode starts from a fresh copy. - Seed any simulated actors, such as an approver who replies after a delay, so replays match.

**Observation and actions**

- Return records the actor is entitled to see. Hide validity labels and the answer key. - Log every tool call with arguments, simulated time and the resulting state version.

**State diff**

- At episode end, compute a diff between the starting snapshot and final state, limited to `request`, `decision_log`, `assignment` and downstream tables such as `payment_hold`. - Express it as added, changed and removed rows keyed by primary key, so graders compare structure and not text.

**Grader hooks**

- `on_step(state, action)` runs the policy checker and appends a `valid` flag to a private log. - `on_end(diff, trace)` returns outcome, process and penalty scores as separate fields. - Task metadata lists accepted end states, so alternate valid routes need no code change.

Keep graders deterministic and free of model calls where possible. If you use a model-based judge for free-text escalation reasons, keep it separate from the state score and spot check it by hand.

Ask in a pilot whether the release ships reference grader code or only the answer key, and which format suits your harness.

Use immutable snapshots and keyed diffs so every episode is reproducible and graders compare structure instead of text.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Delegated approval