Skip to content
MirrorFirm

How do evaluation hooks and environment resets work for escalation runs?

A simulated environment can reset to a seeded state per task, log every tool call and state change, and expose hooks for graders to read the final state and the event trail.

A simulated escalation environment is designed so that a run can be repeated and graded without manual inspection.

**Reset.** Each task references a `starting_state_ref`. A reset call, such as `reset(task_id, seed)`, restores cases, reviewers, policies and queues to that state. Seeds control any simulated reviewer behaviour, for example whether a reviewer returns a case for information.

**Action log.** Every tool call is written with `run_id`, `step`, `tool`, `arguments`, `result` and `error_code`. State changes are written to the same event stream as the environment's own audit events, so the two can be compared.

**Grader hooks.** After a run, a grader can read three things:

- the final state of the case, assignee and escalation records - the ordered event trail - the task's `expected` block

The grader emits per-criterion results such as `trigger`, `target`, `handover` and `boundary`, along with a rework count.

**Simulated reviewers.** Rule-based reviewers can accept, return or reject based on the matrix, which keeps outcomes deterministic. If you want less predictable behaviour, scope that explicitly and keep the seed fixed in regression tests.

The output format for results, and whether hooks are exposed as an HTTP API or a library, can be settled in a pilot to fit your test harness.

Deterministic resets and full action logs make runs repeatable and comparable across agent versions.

Audience:
Agent developers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Credit escalation