Skip to content
MirrorFirm

Which reset and evaluation hooks make correction runs reproducible?

Versioned resets, structured action traces and isolated evaluator outputs let resellers distribute environments whose results customers can reproduce and investigate.

A simulated environment can expose `reset_case(case_id, seed, release_version)` and return an `initial_state_hash`. Reset semantics should cover records, pending events, simulated time and role context. Restoring ledger rows alone is insufficient if an old background event can still alter the next run.

Actions can emit trace records containing `run_id`, `step_id`, `tool_name`, `arguments_digest`, `result_code` and `state_hash_after`. An evaluator can then associate a failed check with the action that introduced the relevant state change.

A separate evaluation hook could return `check_id`, `expected_condition`, `observed_condition` and `failure_record_refs`. Keep protected answers and evaluator-only state outside the agent’s accessible tools. Otherwise, the evaluation interface itself can disclose the solution.

Resellers can distribute a minimal replay specification alongside the environment package. It should identify which sources of nondeterminism are controlled and which remain outside scope. Customers can use matching release versions and seeds to investigate differences between agents, while state hashes help detect a reset or packaging mismatch before interpreting the results.

Make customer evaluation runs attributable to a specific environment, case version and starting state.

Audience:
Licensed resellers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Ledger adjustment