Skip to content
MirrorFirm

What reset and evaluation hooks does a case-escalation environment need for reliable grading?

Reliable grading needs deterministic reset, scripted senior responses, an append-only audit log and a state diff between the start snapshot and the final state.

Reliable grading depends on four hooks.

**Reset.** Load the tables once into a snapshot, then restore it before each episode. Task definitions apply small overrides to the starting state, such as setting a senior role to absent. Reset should clear audit events and any agent-created rows.

**Scripted senior responses.** Give each task a `senior_policy` field such as `approve`, `amend_amount:70000`, `decline` or `return_for_info`. When the agent calls `escalate`, the environment applies the policy deterministically, optionally after a simulated delay step. Fixed seeds keep runs comparable.

**Append-only audit log.** Every tool call writes an `audit_events` row with `actor_role`, `action`, `args_hash` and `ts`. Graders read this to find unauthorised decisions and tool calls made in the wrong order.

**State diff.** At episode end, compute the difference between the snapshot and the final state, keyed by table and primary key. Compare it with `expected_end_state` in the task file, allowing listed fields to vary, such as free-text note bodies.

**Grader outputs:** a JSON object per episode with component scores for routing, target, state, note and follow-through, plus the failure-mode labels triggered.

Run a scripted reference agent through every task first. If it cannot pass, the task or the labels need repair before any model is evaluated.

Deterministic reset plus a state diff and audit log let you grade by what changed, not by what the agent said.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Credit escalation