Skip to content
MirrorFirm

Which environment hooks make handover evaluations reproducible?

Reproducible evaluation needs versioned starting state, recorded tool interactions and explicit assertions that connect agent actions to observable changes in the simulated workflow.

A simulated handover evaluation should identify its starting state with `snapshot_id`, `scenario_version`, `policy_version` and a seed where randomness is used. Reset should restore records, queues, pending events and action history. Restoring only the main case can leave residual state that changes the next run.

Each tool interaction can emit `run_id`, `action_id`, `actor_role`, `tool_name`, `input`, `result` and `state_version`. Evaluation hooks can attach assertions to meaningful transitions: a dependency remains unresolved, a required acknowledgement exists or an action altered a record outside the agent's authority.

A buyer can ask for separate outcome checks and trace checks. Outcome checks inspect the final business state. Trace checks inspect how the agent reached it, including invalid attempts and unnecessary repeated actions. This distinction matters when an agent eventually succeeds after causing avoidable disruption.

Traces could be exported as JSONL, with aggregate results in Parquet for analysis. The evaluator configuration should be versioned alongside the scenario, so changes in a score can be traced to model behaviour, environment changes or revised assessment rules.

Make failed handovers replayable and explainable through environment evidence.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Cross-team handover