Skip to content
MirrorFirm

Which reset and evaluation hooks make customer change episodes reproducible?

Reproducible episodes need controlled resets, event scheduling and observable checks, allowing builders to distinguish agent behaviour from incidental differences in environment execution.

Define an episode manifest containing `scenario_id`, `schema_version`, `policy_version`, `initial_snapshot_id` and `seed`. The seed should control documented randomness such as response selection or worker scheduling. It should not stand in for missing records of external effects.

A reset hook should restore mutable records, pending jobs, tool session state and the simulated clock. An evaluation hook can return named checks with `passed`, `evidence_refs` and an explanation. Record each tool call with its inputs, result, event sequence number and relevant state version.

MirrorFirm could generate manifests, initial snapshots and expected checks for implementation in a buyer's runtime. Builders should keep evaluator-only assertions outside the agent's tool responses. A replay mode can feed recorded external responses back into the same action trace, while a fresh run tests the agent under the original conditions.

Verify reset by running a known sequence, resetting and confirming that no earlier mutation remains observable. When a failure occurs, compare the replay with the fresh run. This helps locate nondeterminism or adapter defects before attributing the result to agent reasoning.

Capture enough execution context to replay failures and identify whether the agent, adapter or evaluator caused them.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Customer change handling