Skip to content
MirrorFirm

How should delivery formats, versioning and data splits be agreed for a pilot?

Lists delivery contract items to agree early: file layout, manifest, published schema, release versioning, dev and held-out splits, tool definitions, reset endpoint and arrival checks.

Agree the delivery contract early so the sample drops into your harness without rework.

**Files.** Entities as JSONL or Parquet, one file per type, plus `policies.jsonl`, `tasks.jsonl` and `expected_state.jsonl`. Ask for a `manifest.json` listing each file, its record count, schema version and the seed ids it covers.

**Schema.** Ask for a published schema, such as JSON Schema for the JSONL files, so you can validate on load. Request that enumerations such as status values and obligation kinds are listed in one place.

**Versioning.** Each release carries a `release_id` and `schema_version`. If a field changes, the version changes. Pin your harness to a version so a re-run is comparable with earlier scores.

**Splits.** Tasks can carry a `split` label such as `dev` or `heldout`, so training and evaluation material stay separate. Ask how splits are generated and whether near-duplicate parties can cross a boundary.

**Tool interface.** If you want a simulated environment rather than files, ask for tool definitions as JSON schemas with names, arguments, return shapes and error codes, plus a reset call that accepts a `seed_id`.

**Checks on arrival.** Validate against the schema, resolve every reference, confirm the manifest counts, and run the grader on a known-correct trajectory before any agent run.

Treat this list as a conversation starter. Final formats, field lists and interfaces are settled when the pilot is scoped.

Agreeing manifest, schema, version pinning and splits up front lets a sample drop into your harness and keeps later scores comparable.

Audience:
Environment builders
Content type:
Integration
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Account setup