Which evaluation hooks make failures reproducible and diagnosable?
Replayable traces can connect observations, actions and state changes to hidden assertions, helping builders diagnose failures without exposing expected answers to agents.
Emit a trace record for each observation and attempted action. Useful fields include `episode_id`, `step_id`, `simulation_time`, `action_name`, `input_digest`, `result_code` and references to the state before and after execution. Preserve rejected actions as well as accepted ones.
Evaluation assertions should read authoritative environment state through a separate interface. An assertion might check that no outbound message references an obligation after its valid closure. Its identifier and result belong in the evaluator output, not in the agent's tool response.
For replay, retain the initial snapshot reference, environment version, generation seed where applicable, and ordered external events. A seed alone may be insufficient if tool behaviour or policy interpretation has changed. Versioned snapshots and event definitions make that dependency explicit.
MirrorFirm could supply generated traces for expected and failing paths alongside assertion definitions. Builders can export step events as JSONL and aggregate outcomes in Parquet. A replay hook should stop at the first divergent state, allowing a reviewer to identify whether the failure came from an observation, an agent choice or the environment implementation.
Capture enough execution evidence to reproduce a failure and identify its causal step.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should buyers run before using a generated task pack?
- Which evaluation hooks expose duplicate actions and unreliable recovery?
- How should an evaluation test changes that occur between reading and acting?
- How should follow-up quality be scored when several responses are acceptable?
- How can buyers tell whether an agent follows policy instead of superficial cues?
- What should we inspect before trusting an environment's evaluation results?