How should a pilot manage versions across downloads and MCP access?
A pilot can use explicit package identities and compatibility checks to keep downloaded tasks and MCP-delivered material aligned with the evaluator.
Ask for a delivery manifest that identifies `package_id`, `release_id`, `schema_version`, `task_id` and `rubric_version`. Where the same material is offered through downloadable exports and MCP resources, request a documented mapping between those identifiers and the corresponding resource references.
The runner should record the manifest with its results. Content digests can help detect an unexpected change between retrieval and execution. A task retrieved from a public endpoint should not silently replace the version used in an earlier comparison.
Scope a compatibility check before connecting the full evaluation runner. It can load a task, resolve its referenced records, validate required fields and confirm that agent-visible content excludes evaluator-only material. If the pilot includes an executable environment, check that its snapshot identifier matches the task's declared starting state.
Ask how corrections will be represented and how retired versions can be identified. These are requirements to discuss, not assumed service commitments. A useful first integration step is a small contract check against one agreed package, with clear failure messages for missing resources, incompatible schemas and mismatched rubric versions.
Make each evaluation run attributable to a specific package, schema and scoring definition across delivery methods.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What should a buyer include in a practical pilot request?
- What licence questions should buyers settle before using or redistributing generated packs?
- What integration contract should a pilot establish before agents consume a release?
- How can a sample reveal a missing scope decision before a larger release is defined?
- How should a team bound its first policy-propagation pilot?
- What integration contract should be settled before generating a pilot package?