Skip to content
MirrorFirm

Which reset and evaluation hooks make routing experiments reproducible?

Explicit reset inputs and structured trajectory hooks let builders replay routing episodes and attribute outcomes to observable decisions and state changes.

A reset contract can accept `scenario_id`, `fixture_revision`, `random_seed` and `policy_version`, then return an `episode_id` and the initial observation. The environment should reset pending events, tool-side caches and simulated time as well as request records; residual state can change the next run’s outcome.

Evaluation hooks can capture `before_action`, `after_action` and `episode_end`. Each captured item can include an action identifier, simulation timestamp, visible tool result and state-diff reference. Restricted evaluator state should remain outside the observation returned to the agent.

A terminal evaluator can produce separate fields for destination eligibility, unresolved obligations and prohibited transitions. Keeping these components separate helps a builder distinguish a valid final route from a trajectory that reached it through an invalid action.

MirrorFirm could provide generated fixtures with replay metadata and expected evaluator inputs. A buyer can check reproducibility by resetting the same episode and replaying the same actions against the same event schedule. Where runtime generation introduces variation, the integration should record the realised responses as replay artefacts rather than assuming a seed alone explains every result.

Make every score traceable to a reproducible episode and its recorded transitions.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Service routing