What should we ask for in a sample or pilot release for set-up?
A practical list of items to request in a sample, including schemas, a small case set, answer keys and review evidence, so you can judge fit before committing further.
A sample should let your engineers and reviewers reach a view without extra meetings. Ask for the following.
- **Schemas and a data dictionary.** Every record type, field meaning, enumeration and link rule. - **A small, varied case set.** Include clean, near-match, incomplete and refuse-correctly cases, not only easy ones. - **The policy text.** Both the plain-language version and any structured form, with the cases that depend on each rule. - **A tool manifest.** Action names, arguments and return shapes, plus notes on how they map to your agent framework. - **Answer keys and a grader.** Expected end states, failure-mode tags and a runnable check, kept separate from agent-visible files. - **Reset and seed behaviour.** Confirmation that a run can be reproduced, and how to regenerate with a different seed. - **A statement of generation inputs and controls.** What informed the release and what disclosure checks were applied, so your reviewers can assess it independently. - **Open limitations.** A written list of what the sample does not cover, such as products, regions or fault types.
Then decide in advance who reviews which part, and what result would lead you to widen the pilot, adjust it or stop.
Requesting schemas, answer keys and review evidence with the sample lets you judge fit and risk, not just skim a few rows.
- Audience:
- Agent developers
- Content type:
- Checklist
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot look like for a client and matter opening agent?
- What licence and usage limits should we ask about before starting?
- How should we scope a first pilot for set-up agents?
- How do we wire a pilot release into our agent harness on day one?
- How should delivery formats, versioning and data splits be agreed for a pilot?
- What should I know about licensing, limits and getting started?