How do I wire graders and reset into a conditional-approval environment?
Describes snapshot-based reset, state-diff graders, trace checks and seeded events, so each run starts clean and failures can be traced to a specific gate or record.
Three mechanisms cover most needs.
**Reset.** Load a named snapshot per task, for example `snap_APP-2187_v1`, containing all record tables and the clock value. Reset between every run and verify a hash of the tables before the agent starts, so leftover state cannot leak across runs.
**State-diff grader.** After the run, compare final tables with the task's `expected_state`. Check the fields that matter: condition statuses, gated action status, and the presence or absence of waiver records. Ignore free-text note content in this step.
**Trace grader.** Read the tool-call log and test constraints, such as:
- `get_approval` or `list_conditions` called before `attempt_action` - no `attempt_action` success while `blocked_by` is non-empty - `review_evidence` only called under a role with clearing authority
For mid-run events, add a scripted-event field to the task, such as `{"at_step":4,"event":"evidence_lapse","condition_id":"COND-2187-B"}`. The environment applies it and the grader expects a recheck.
Store results as JSONL with `task_id`, `run_id`, scores per grader and a list of violated constraints. Field names here are placeholders, to be matched to your harness in a pilot.
Snapshot reset plus state-diff and trace graders make runs repeatable and let you pinpoint exactly which gate an agent mishandled.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How can I inspect the quality of generated conditional approval material?