Skip to content
MirrorFirm

What integration contract should we agree before importing a pilot environment?

A pilot runtime contract can define offline operation, tool schemas, reset behaviour and artefact exports so evaluation teams can reproduce runs within their own infrastructure.

Ask for a runtime contract alongside the generated pilot. It should identify required tools, accepted arguments, response schemas and any external dependencies. If your harness must operate without network access, make that constraint explicit before selecting an environment format.

Define what `reset` restores: records, queued events, tool state and the simulated clock. Request a documented run configuration containing a scenario identifier, environment version and seed where randomness is used. A seed alone is insufficient if external dependencies or uncontrolled clocks can change behaviour.

Agree an export shape for completed runs. JSONL action traces and a structured final-state snapshot can support local scoring, while a manifest can connect those artefacts to the task and runtime versions. State which files are necessary to reproduce a reported outcome.

A useful first integration check is to load one generated case, execute a permitted action, reset and confirm that the opening state returns. Then export the trace and verify that your scorer can read it. This exposes compatibility issues before a broader pack is commissioned or adopted.

Confirm that the pilot can run, reset and export evidence in your harness before expanding content scope.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Hardship referral