Skip to content
MirrorFirm

Which evaluation hooks make a handover episode auditable and repeatable?

Replayable traces connect observations, actions and state changes, allowing builders to explain a score and reproduce failures under the same environment configuration.

An evaluation trace should connect what the agent saw, what it attempted and what the environment changed. Useful fields include `episode_id`, `step_id`, `observation_ref`, `tool_call`, `tool_result`, `state_diff_ref` and `rule_version`. Store the evaluator's findings separately from agent-visible observations.

A post-transition hook can inspect invariants after every accepted action. A terminal hook can assess the final business outcome. Keeping these hooks separate helps identify an invalid intermediate action even when a later action repairs the final state.

For reproducibility, record the initial snapshot identifier, environment version, random seed and scheduled external events. Reset should restore the simulated clock and pending event schedule as well as business records. Reusing a snapshot while retaining a delayed notification can change the next run unexpectedly.

The scorer can emit structured reasons such as `dependency_detached` or `unsupported_closure`, with references to the relevant transition. Avoid returning these private scoring reasons through ordinary agent tools during evaluation.

Buyers can request a replay demonstration in which the stored trace reproduces the same outcome and failure reason. Where external components prevent exact replay, document that boundary and preserve their recorded responses.

Capture enough execution context to diagnose why an episode passed or failed without exposing hidden answers to the agent.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Cross-team handover