What trace format and grader hooks make conditional-approval evaluation automatic?
Log each tool call with a state hash, then score forbidden calls, gated actions, unread evidence, final state and stated reason against task ground truth.
Evaluation works best when the harness records enough to score both outcome and process.
**Trace format.** Log each tool call as a JSONL line with `run_id`, `task_id`, `step`, `tool`, `arguments`, `result_status` and `state_hash_after`. The state hash lets a grader detect which step changed the environment.
**Grader inputs.** The grader reads the task's `expected_actions`, `forbidden_actions`, `reason_code` and the final environment snapshot.
**Checks to run automatically:**
- any forbidden tool call, flagged with the step number - gated actions taken while `conditions.state` was not in an allowed set - condition states changed without a preceding read of the linked evidence - final condition and work item states against the expected snapshot - the stated reason matched against `reason_code`
**Result record.** Write one line per task with `outcome`, `premature_action`, `unnecessary_hold`, `wrong_write` and `reason_match` as booleans, plus a free-text `reviewer_note`.
**Reset.** Each task references an `initial_state_ref`. Restore it before every run and, if your agent keeps memory, clear that too. Run each task several times to see variance.
**Time.** If tasks depend on expiry, expose a controllable clock so a run can be repeated deterministically.
Define the exact fields with MirrorFirm in a pilot so they map to your harness.
A step-level trace with state hashes lets graders pinpoint which action broke the gate, so failures lead to specific fixes.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?
- How can I inspect the quality of generated conditional approval material?