Which reset and evaluation hooks make routing experiments reproducible?
Explicit reset inputs and structured trajectory hooks let builders replay routing episodes and attribute outcomes to observable decisions and state changes.
A reset contract can accept `scenario_id`, `fixture_revision`, `random_seed` and `policy_version`, then return an `episode_id` and the initial observation. The environment should reset pending events, tool-side caches and simulated time as well as request records; residual state can change the next run’s outcome.
Evaluation hooks can capture `before_action`, `after_action` and `episode_end`. Each captured item can include an action identifier, simulation timestamp, visible tool result and state-diff reference. Restricted evaluator state should remain outside the observation returned to the agent.
A terminal evaluator can produce separate fields for destination eligibility, unresolved obligations and prohibited transitions. Keeping these components separate helps a builder distinguish a valid final route from a trajectory that reached it through an invalid action.
MirrorFirm could provide generated fixtures with replay metadata and expected evaluator inputs. A buyer can check reproducibility by resetting the same episode and replaying the same actions against the same event schedule. Where runtime generation introduces variation, the integration should record the realised responses as replay artefacts rather than assuming a seed alone explains every result.
Make every score traceable to a reproducible episode and its recorded transitions.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can buyers test whether an agent uses the right evidence to choose a destination?
- Which evaluation hooks reveal whether a routing action preserved the required handover?
- What should buyers inspect before using a generated routing corpus?
- How can buyers detect answer leakage in generated routing evaluations?
- How should evaluation handle a destination that changes while the agent is working?
- What should buyers check in generated content before trusting agent scores?