Skip to content
MirrorFirm

Which evaluation hooks make failures reproducible and diagnosable?

Replayable traces can connect observations, actions and state changes to hidden assertions, helping builders diagnose failures without exposing expected answers to agents.

Emit a trace record for each observation and attempted action. Useful fields include `episode_id`, `step_id`, `simulation_time`, `action_name`, `input_digest`, `result_code` and references to the state before and after execution. Preserve rejected actions as well as accepted ones.

Evaluation assertions should read authoritative environment state through a separate interface. An assertion might check that no outbound message references an obligation after its valid closure. Its identifier and result belong in the evaluator output, not in the agent's tool response.

For replay, retain the initial snapshot reference, environment version, generation seed where applicable, and ordered external events. A seed alone may be insufficient if tool behaviour or policy interpretation has changed. Versioned snapshots and event definitions make that dependency explicit.

MirrorFirm could supply generated traces for expected and failing paths alongside assertion definitions. Builders can export step events as JSONL and aggregate outcomes in Parquet. A replay hook should stop at the first divergent state, allowing a reviewer to identify whether the failure came from an observation, an agent choice or the environment implementation.

Capture enough execution evidence to reproduce a failure and identify its causal step.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Overdue follow up