What integration contract should be agreed before connecting a pilot to an evaluation harness?
A portable pilot interface should define episode loading, tool discovery and result export so evaluation teams can reproduce runs across release changes.
Ask for an integration contract covering how your harness discovers tasks, loads an episode and collects results. A manifest can identify `release_id`, `schema_version`, `episode_id`, `policy_bundle_id` and the supported tool interface. Keep these identifiers in exported results so a score can be traced to the exact material used.
For a downloadable pack, define file layout, encoding and how task records reference supporting tables. For MCP access, specify the resources and tool descriptions your harness needs, including pagination, error shapes and whether retrieval returns a fixed release or changing content. These are interface requirements to scope, not assumptions about current endpoints.
Agree reset semantics explicitly. A reset response should identify the restored episode and initial-state fingerprint, allowing the harness to check that a run began from the intended state. Define how stochastic elements are seeded or recorded.
Finally, request a result export containing termination status, action trace references and constraint-level evaluation outcomes. Start by loading and resetting a sample episode through your own harness; that establishes whether the proposed interface supports a repeatable pilot.
Agree the execution interface early so integration effort produces reproducible evidence rather than an environment-specific demonstration.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- How can a pilot cover closed accounting periods without modelling an entire finance function?
- What licence questions should we resolve before using generated material in an agent pipeline?
- What should a correction pilot brief specify before generation begins?
- What integration details should we settle before loading a pilot into our agent harness?
- What is a useful first scope for an agent correction pilot?
- How can a pilot expose a scope boundary without implementing an entire finance system?