Skip to content
MirrorFirm

How should a pilot delivery be structured so my team can load and run it?

A pilot can ship a manifest, schemas, data files, versioned policy, tasks with gold outcomes, environment definitions and documentation, with stable identifiers, immutable snapshots and labelled splits.

A pilot delivery can be organised so your team loads it, resets it and runs evaluations without custom conversion. Ask for the following structure and confirm it in the scoping discussion.

- `manifest.json` listing entities, file paths, formats, schema versions and the release identifier - `schemas/` with a JSON Schema per entity, including enumerations for reason codes and outcomes - `data/` with JSONL for text-heavy entities and Parquet for larger relational tables - `policy/` with versioned authority matrices and policy text, each with an effective date - `tasks/` with task definitions, seed snapshot references and gold end states - `env/` with tool definitions, the reset procedure and the simulated reviewer rules - `docs/` with a data dictionary and a construction note

Integration detail worth confirming: identifiers are stable across files, so `APP-2187` means the same thing in every table; snapshot identifiers are immutable, so a rerun months later starts from identical state; and splits are labelled, for example `split: train | dev | heldout`, so evaluation items stay apart from training items.

For hooks, ask for a function that loads a snapshot into your own store and a function that exports the final state and event log after a run. Where your stack differs, such as a SQL store rather than files, say so early so the delivery format can be agreed.

A predictable folder structure, stable identifiers and labelled splits let your team load, reset and evaluate without conversion work.

Audience:
Model labs
Content type:
Integration
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Credit escalation