Skip to content
MirrorFirm

What should a pilot delivery package contain so it is easy to load and rerun?

A suggested pilot package layout with a manifest, case files, policies, withheld labels, graders and environment definitions, plus advice on stable identifiers and acceptance steps.

A pilot delivery should be easy to load, inspect and rerun. A sensible package includes:

- `manifest.json`: release name, `schema_version`, `generator_version`, file list and checksums, so you can confirm you hold what was sent. - `cases.jsonl` and matching Parquet tables for `approvals`, `conditions`, `evidence`, `waivers`, `work_items` and `events`. - `policies/`: Markdown or JSON policy text with a `policy_version` referenced by each case. - `labels.jsonl`: expected action, expected final state and open condition IDs, kept separate from the case files so it can be withheld from models. - `graders/`: check functions that take a run trace and return pass or fail per failure mode. - `env/`: tool schemas, an initial state snapshot per case and a `reset(case_id, seed)` entry point, if an environment is in scope.

Ask that identifiers be stable between versions, so `APP-2187` in one drop refers to the same case in the next, and that any regenerated case carries a new `case_id` and a change note.

Pilot runs can start locally by loading the JSONL, replaying the timeline and calling the graders. Wire it into your own harness after the sample is reviewed. Everything delivered is generated, so include a grounding and disclosure review in your acceptance steps.

A manifest, separated labels, graders and stable identifiers make a pilot easy to load, verify and rerun across releases.

Audience:
Model labs
Content type:
Integration
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Approval conditions