Skip to content
MirrorFirm

Which environment hooks make downstream evaluations reproducible?

Versioned reset states and structured action traces let customers reproduce a task and inspect whether apparently successful outcomes depended on invalid operations.

A simulated environment could expose `reset(snapshot_id, seed)`, `get_observation(role)` and `step(action)`. The reset response should identify the scenario revision, policy revision and tool schema revision. A seed is useful only alongside fixed generator and environment versions; it should not be presented as sufficient by itself.

Each action trace could record `run_id`, `step_id`, `action_type`, `target_id`, `result_code` and state changes. Evaluator-only hooks could inspect invariants such as whether a hold was removed before its required approval existed. These hooks should remain outside the agent's observation surface.

For a reseller, the interface contract should specify whether reset clears queued notices, pending approvals and simulated clock changes. Residual state can make the same task easier or harder on a later run.

Customers could export traces as JSONL and compare final state against reference conditions. The evaluation should retain failed actions as well as successful ones: reaching the target state after an unauthorised attempt is different from completing the workflow correctly throughout. This distinction can be preserved even when customers supply their own scoring layer.

Provide a reproducible evaluation interface whose evidence survives integration into a reseller's own testing platform.

Audience:
Licensed resellers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Policy change propagation