What should a pilot delivery package contain so it is easy to load and rerun?
A suggested pilot package layout with a manifest, case files, policies, withheld labels, graders and environment definitions, plus advice on stable identifiers and acceptance steps.
A pilot delivery should be easy to load, inspect and rerun. A sensible package includes:
- `manifest.json`: release name, `schema_version`, `generator_version`, file list and checksums, so you can confirm you hold what was sent. - `cases.jsonl` and matching Parquet tables for `approvals`, `conditions`, `evidence`, `waivers`, `work_items` and `events`. - `policies/`: Markdown or JSON policy text with a `policy_version` referenced by each case. - `labels.jsonl`: expected action, expected final state and open condition IDs, kept separate from the case files so it can be withheld from models. - `graders/`: check functions that take a run trace and return pass or fail per failure mode. - `env/`: tool schemas, an initial state snapshot per case and a `reset(case_id, seed)` entry point, if an environment is in scope.
Ask that identifiers be stable between versions, so `APP-2187` in one drop refers to the same case in the next, and that any regenerated case carries a new `case_id` and a change note.
Pilot runs can start locally by loading the JSONL, replaying the timeline and calling the graders. Wire it into your own harness after the sample is reviewed. Everything delivered is generated, so include a grounding and disclosure review in your acceptance steps.
A manifest, separated labels, graders and stable identifiers make a pilot easy to load, verify and rerun across releases.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot review look like for a referral-processing agent team?
- How is generated conditional-approval material licensed, and what limits apply?
- What should a pilot request include so the first sample can be judged quickly?
- How should I scope a first pilot for conditional-approval material?
- How do I wire a pilot sample into my agent harness for a first run?
- What should I ask for in a first sample or pilot?