How should delivery formats, versioning and data splits be agreed for a pilot?
Lists delivery contract items to agree early: file layout, manifest, published schema, release versioning, dev and held-out splits, tool definitions, reset endpoint and arrival checks.
Agree the delivery contract early so the sample drops into your harness without rework.
**Files.** Entities as JSONL or Parquet, one file per type, plus `policies.jsonl`, `tasks.jsonl` and `expected_state.jsonl`. Ask for a `manifest.json` listing each file, its record count, schema version and the seed ids it covers.
**Schema.** Ask for a published schema, such as JSON Schema for the JSONL files, so you can validate on load. Request that enumerations such as status values and obligation kinds are listed in one place.
**Versioning.** Each release carries a `release_id` and `schema_version`. If a field changes, the version changes. Pin your harness to a version so a re-run is comparable with earlier scores.
**Splits.** Tasks can carry a `split` label such as `dev` or `heldout`, so training and evaluation material stay separate. Ask how splits are generated and whether near-duplicate parties can cross a boundary.
**Tool interface.** If you want a simulated environment rather than files, ask for tool definitions as JSON schemas with names, arguments, return shapes and error codes, plus a reset call that accepts a `seed_id`.
**Checks on arrival.** Validate against the schema, resolve every reference, confirm the manifest counts, and run the grader on a known-correct trajectory before any agent run.
Treat this list as a conversation starter. Final formats, field lists and interfaces are settled when the pilot is scoped.
Agreeing manifest, schema, version pinning and splits up front lets a sample drop into your harness and keeps later scores comparable.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot look like for a client and matter opening agent?
- What licence and usage limits should we ask about before starting?
- How should we scope a first pilot for set-up agents?
- What should we ask for in a sample or pilot release for set-up?
- How do we wire a pilot release into our agent harness on day one?
- What should I know about licensing, limits and getting started?