Which reset and evaluation hooks make customer change episodes reproducible?
Reproducible episodes need controlled resets, event scheduling and observable checks, allowing builders to distinguish agent behaviour from incidental differences in environment execution.
Define an episode manifest containing `scenario_id`, `schema_version`, `policy_version`, `initial_snapshot_id` and `seed`. The seed should control documented randomness such as response selection or worker scheduling. It should not stand in for missing records of external effects.
A reset hook should restore mutable records, pending jobs, tool session state and the simulated clock. An evaluation hook can return named checks with `passed`, `evidence_refs` and an explanation. Record each tool call with its inputs, result, event sequence number and relevant state version.
MirrorFirm could generate manifests, initial snapshots and expected checks for implementation in a buyer's runtime. Builders should keep evaluator-only assertions outside the agent's tool responses. A replay mode can feed recorded external responses back into the same action trace, while a fresh run tests the agent under the original conditions.
Verify reset by running a known sequence, resetting and confirming that no earlier mutation remains observable. When a failure occurs, compare the replay with the fresh run. This helps locate nondeterminism or adapter defects before attributing the result to agent reasoning.
Capture enough execution context to replay failures and identify whether the agent, adapter or evaluator caused them.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?