How could a pilot be delivered so runs are repeatable and resettable?

A pilot is easier to evaluate and repeat when it ships with a manifest and fixed seeds. A delivery could include the following.

- **Manifest.** A `manifest.json` listing every file, its schema version, a checksum, the split it belongs to and the release identifier. - **Seeds and variants.** Each task names the generation seed and the policy variant it came from, so a regenerated batch can be compared with the previous one. - **Snapshots.** Named start states, such as `snap-pilot-0009-a`, held as Parquet tables that an environment loads in full. - **Reset.** A documented command or tool action that restores a snapshot before each run. Without it, one run's approvals contaminate the next. - **Tool schema.** JSON definitions for actions such as `get_policy`, `lookup_delegation`, `check_availability`, `route_request`, `record_approval` and `escalate`, with argument types and error responses. - **Changelog.** Notes of what changed between samples, so corrections requested after review are traceable.

If your team uses the Model Context Protocol, the tool schema could be exposed through a server so agents call the same actions in training and in evaluation. Scope that with MirrorFirm as part of the pilot rather than assuming it.

Ask also how long snapshots are retained and how a version is frozen for a published result. Keep a copy of the manifest with every score you report, so results can be tied to the exact material used.