Skip to content
MirrorFirm

What integration contract should be agreed before connecting a pilot to an evaluation harness?

A portable pilot interface should define episode loading, tool discovery and result export so evaluation teams can reproduce runs across release changes.

Ask for an integration contract covering how your harness discovers tasks, loads an episode and collects results. A manifest can identify `release_id`, `schema_version`, `episode_id`, `policy_bundle_id` and the supported tool interface. Keep these identifiers in exported results so a score can be traced to the exact material used.

For a downloadable pack, define file layout, encoding and how task records reference supporting tables. For MCP access, specify the resources and tool descriptions your harness needs, including pagination, error shapes and whether retrieval returns a fixed release or changing content. These are interface requirements to scope, not assumptions about current endpoints.

Agree reset semantics explicitly. A reset response should identify the restored episode and initial-state fingerprint, allowing the harness to check that a run began from the intended state. Define how stochastic elements are seeded or recorded.

Finally, request a result export containing termination status, action trace references and constraint-level evaluation outcomes. Start by loading and resetting a sample episode through your own harness; that establishes whether the proposed interface supports a repeatable pilot.

Agree the execution interface early so integration effort produces reproducible evidence rather than an environment-specific demonstration.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Ledger adjustment