How should a pilot delivery be structured so my team can load and run it?
A pilot can ship a manifest, schemas, data files, versioned policy, tasks with gold outcomes, environment definitions and documentation, with stable identifiers, immutable snapshots and labelled splits.
A pilot delivery can be organised so your team loads it, resets it and runs evaluations without custom conversion. Ask for the following structure and confirm it in the scoping discussion.
- `manifest.json` listing entities, file paths, formats, schema versions and the release identifier - `schemas/` with a JSON Schema per entity, including enumerations for reason codes and outcomes - `data/` with JSONL for text-heavy entities and Parquet for larger relational tables - `policy/` with versioned authority matrices and policy text, each with an effective date - `tasks/` with task definitions, seed snapshot references and gold end states - `env/` with tool definitions, the reset procedure and the simulated reviewer rules - `docs/` with a data dictionary and a construction note
Integration detail worth confirming: identifiers are stable across files, so `APP-2187` means the same thing in every table; snapshot identifiers are immutable, so a rerun months later starts from identical state; and splits are labelled, for example `split: train | dev | heldout`, so evaluation items stay apart from training items.
For hooks, ask for a function that loads a snapshot into your own store and a function that exports the final state and event log after a run. Where your stack differs, such as a SQL store rather than files, say so early so the delivery format can be agreed.
A predictable folder structure, stable identifiers and labelled splits let your team load, reset and evaluate without conversion work.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What licensing questions should I raise for generated escalation material?
- What might a first escalation pilot show an agent team?
- What should my team prepare to start an escalation pilot?
- How should I scope a first escalation pilot with MirrorFirm?
- What should I ask for in an escalation sample or pilot?
- What licensing, limits and boundaries should a buyer ask about before a pilot?