What delivery formats and environment setup should a pilot specify?
Lists the file formats, versioning, validation scripts, environment interface and reset expectations a pilot can specify so the output drops into an existing evaluation stack.
Put integration needs into the pilot brief so the first delivery runs in your stack without rework.
**Data files.** Choose JSONL for record-by-record loading, Parquet for analytics, or both. Ask for one file per record family, such as authority, delegations, requests, routing and gold, with documented primary and foreign keys.
**Versioning.** Request a release identifier in each file and a manifest listing files, schema versions and record counts, so you can detect drift between deliveries.
**Validation.** Ask for a schema definition, for instance JSON Schema, and a script or checklist to verify keys, date ranges and limit arithmetic. Run it in your own pipeline, not only theirs.
**Task pack layout.** Each task bundles a prompt, the record references it needs and a gold entry with condition tags.
**Environment option.** If you want a simulated environment, specify the tool interface (for example function-call signatures or an MCP-style tool list), whether write tools enforce authority or only log attempts, and the simulated clock behaviour.
**Reset and runs.** Agree on snapshot loading, seeded randomness and an append-only event log, so results are repeatable and can be compared across model versions.
**Output.** Define the results format your graders will emit, including per-condition fields.
The content is generated to your specification. Confirm formats and reset behaviour on the sample, along with grounding and disclosure risk.
Specifying formats, versioning and reset behaviour at the start prevents rework and lets the pilot output run in your stack on day one.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- How does reviewing a sample change the scope of a freight claims pilot?
- What limits apply when licensing generated approval datasets or environments?
- How do I load a first approval sample into my agent harness?
- How should we scope a first pilot for approval-handling agents?
- What should I ask for in a sample or pilot of approval material?
- What does a scoping session for an approvals pilot produce?