What should I ask for in a first pilot sample request?
A checklist of eight requests to put in writing for a pilot sample, from domain and condition types to splits and licence questions.
When you ask for a pilot sample, put these requests in writing so both sides can judge the result against the same list.
- Name the domain and decision type, such as loan approvals or claims estimates, and keep to one of them at first. - State the condition types you want, and how many conditions a single decision may carry in the sample. - Ask for cases that end in hold, escalation and clean release, and say roughly what mix you want. - Request near-miss evidence that is attached but fails on date, entity, signature or threshold. - Ask for the schema file, a manifest, and expected end states with accepted reason codes for each task. - Say whether you need records only, a task pack, or a simulated environment with tools and reset. - Ask for separate training and held-out splits, divided by decision rather than by individual row. - List the licence questions to settle, including training use, evaluation use and publishing results.
Putting these requests in writing lets both sides judge the sample against the same list.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot review look like for a referral-processing agent team?
- How is generated conditional-approval material licensed, and what limits apply?
- What should a pilot request include so the first sample can be judged quickly?
- How should I scope a first pilot for conditional-approval material?
- How do I wire a pilot sample into my agent harness for a first run?
- What should I ask for in a first sample or pilot?