How should a pilot delivery be packaged, versioned and connected to our harness?
A pilot could be delivered as versioned files with a manifest, seeded generation, a tool schema and reset instructions, so your harness can load, run and re-run it consistently.
Agree the packaging early, because it determines how quickly you can run the first evaluation.
**Files.** JSONL or Parquet per entity, plus `tasks.jsonl` and a `policies` file. Each delivery carries a `manifest.json` with schema version, file list, keys, generation seed and a content hash per file.
**Versioning.** Treat each delivery as immutable. A change, such as a corrected label, becomes a new version with a short change note, so past results remain interpretable. Pin your harness to a version and record it with every run.
**Tool schema.** If you want an environment, ask for a JSON schema for each action, for example `create_facility(approval_id, limit, term_months, repayment_frequency, balloon_amount)` and `hold_file(facility_ref, reason_code, route_to_role)`. Include error responses so the agent sees realistic failures.
**Reset.** A `reset(task_id)` entry point restores the start snapshot. Provide a state hash so you can verify it.
**Evaluation hooks.** Specify where the action log is written, how `grade(run_id)` is called, and which fields are volatile.
**Review loop.** Plan how label disputes and suspected disclosure concerns are raised and resolved. Confirm the details during scoping, not after delivery.
Versioned, seeded and documented deliveries make benchmark results reproducible and let you compare runs across agent versions.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot look like for an agent team in vendor finance?
- What limits apply, and will we receive any real customer records?
- How could a pilot be delivered, versioned and wired into our CI?
- How should a team scope and license a generated booking pack?
- What should we ask for in a sample or pilot of generated booking material?
- How should a sample be delivered, versioned and load-checked in my pipeline?