Skip to content
MirrorFirm

Which environment hooks make interactive evaluations reproducible?

Explicit reset, observation and scoring hooks let resellers distribute interactive tasks whose outcomes can be compared from equivalent starting conditions.

A simulated environment can expose `reset(case_id, seed)`, `observe()`, permitted mutation tools and `score(run_id)`. Reset should restore record versions, queued jobs, simulated time and notification state. Restoring only the customer table can leave later runs contaminated by actions from an earlier attempt.

Supply a manifest containing `environment_version`, `policy_version`, `case_id`, `seed` and `initial_state_digest`. Record tool calls and resulting state transitions under a run identifier. These fields let a downstream buyer reproduce the starting conditions and trace a failed outcome.

Keep scoring assertions outside the agent's observation surface. The agent may need a policy document to decide what to do, but it should not receive the hidden expected action sequence. Score valid final states and required constraints without rejecting an alternative permitted sequence merely because it differs from a reference trace.

For resellers, document which behaviours are deterministic and which depend on seeded events. Include a reset verification check that compares the restored state with the declared initial state. This makes environment packaging part of evaluation quality rather than an undocumented responsibility passed to each downstream customer.

Define a reproducible execution contract so downstream buyers can distinguish agent behaviour from environment variation.

Audience:
Licensed resellers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Customer change handling