How could a pilot delivery be organised so my team can load it quickly?
A pilot can ship as a self-describing bundle with a manifest, schemas, JSONL cases, Parquet tables, policies, tasks, environment seeds and grader files, plus split and feedback conventions.
A pilot delivery can be organised as a small, self-describing bundle so your team can load it without back-and-forth.
- `manifest.json`: version, domain, seed list, file hashes, schema versions and the agreed scope. - `schema/`: JSON Schema for each record type and for each tool action. - `data/cases.jsonl`: one nested case history per line, with submission, checks, actions and outcome. - `data/tables/*.parquet`: flat tables for submissions, cases, rules, queues and links, joined by identifier. - `policies/`: numbered rules with effective dates, in JSON and in readable text. - `tasks/tasks.jsonl`: `task_id`, `seed_id`, instruction, `allowed_tools`, `max_steps`, `reference`. - `env/`: seed snapshots and a reset entry point that takes a `seed_id` and optional perturbations. - `eval/`: grader code or specification, and a results schema.
**Loading.** Read JSONL line by line, validate against the schema, and check file hashes against the manifest before use.
**Splitting.** Ask for train, validation and held-out splits by seed or by rule, so evaluation tasks do not share cases with training material. State which split rule you need.
**Feedback loop.** Return a list of failed tasks and suspected ambiguities using the same `task_id` values, so changes can be targeted.
The content is generated. Include a review of free text for unintended resemblance to real entities in your acceptance steps.
A manifest, schemas and seed-based splits let your team validate, load and give targeted feedback without back-and-forth.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a scoped pilot for a claim lodgement agent look like?
- What should a pilot brief for intake include?
- What should we ask for in an intake sample or pilot?
- How should a team scope a first intake dataset or pilot?
- How do we connect pilot material to our agent harness?
- What delivery layout should I ask for in a pilot so my harness can load it?