What reset and evaluation hooks does a case-escalation environment need for reliable grading?
Reliable grading needs deterministic reset, scripted senior responses, an append-only audit log and a state diff between the start snapshot and the final state.
Reliable grading depends on four hooks.
**Reset.** Load the tables once into a snapshot, then restore it before each episode. Task definitions apply small overrides to the starting state, such as setting a senior role to absent. Reset should clear audit events and any agent-created rows.
**Scripted senior responses.** Give each task a `senior_policy` field such as `approve`, `amend_amount:70000`, `decline` or `return_for_info`. When the agent calls `escalate`, the environment applies the policy deterministically, optionally after a simulated delay step. Fixed seeds keep runs comparable.
**Append-only audit log.** Every tool call writes an `audit_events` row with `actor_role`, `action`, `args_hash` and `ts`. Graders read this to find unauthorised decisions and tool calls made in the wrong order.
**State diff.** At episode end, compute the difference between the snapshot and the final state, keyed by table and primary key. Compare it with `expected_end_state` in the task file, allowing listed fields to vary, such as free-text note bodies.
**Grader outputs:** a JSON object per episode with component scores for routing, target, state, note and follow-through, plus the failure-mode labels triggered.
Run a scripted reference agent through every task first. If it cannot pass, the task or the labels need repair before any model is evaluated.
Deterministic reset plus a state diff and audit log let you grade by what changed, not by what the agent said.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- Which failure modes should an escalation evaluation cover?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?