How do we wire a pilot release into our agent harness on day one?
Load the schemas and tasks, map the tool manifest to your framework, run reset-and-grade on one task, then scale to splits, logging results by failure mode.
Aim to get one task through the full loop before running anything at scale.
**1. Load and validate.** Read the manifest, check file counts and checksums, and validate each record file against its schema. Confirm identifier patterns and date formats match what your tools expect, and note any mapping you need.
**2. Map tools.** Translate the tool manifest into your framework's function-calling format. Keep names and argument types identical, or maintain an explicit mapping table so the audit trail still lines up with the answer key.
**3. Stand up the environment.** Start the simulated systems with the seeded state. Call reset with a task ID and seed, then take a baseline snapshot.
**4. Run one task.** Give the agent the instruction and allowed tools only. Do not expose the answer key, fault plan or grader. Write the run log as JSONL.
**5. Grade.** Take a final snapshot, run the grader, and store its structured output, including failure-mode tags and the honesty flag.
**6. Repeat for a handful of tasks.** Include a clean case, a near-match and a refuse-correctly case. Check that resets leave no residue.
**7. Scale by split.** Run training, validation and held-out sets separately, and keep held-out results out of any tuning loop.
**8. Report back.** Share schema mismatches, unrealistic cases and unclear policies with MirrorFirm so the next release addresses them.
A one-task end-to-end run on day one surfaces schema and tool mismatches early, before you run the full pilot set.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot look like for a client and matter opening agent?
- What licence and usage limits should we ask about before starting?
- How should we scope a first pilot for set-up agents?
- What should we ask for in a sample or pilot release for set-up?
- How should delivery formats, versioning and data splits be agreed for a pilot?
- What should I know about licensing, limits and getting started?