Skip to content
MirrorFirm

What should I ask for in a first pilot sample request?

A checklist of eight requests to put in writing for a pilot sample, from domain and condition types to splits and licence questions.

When you ask for a pilot sample, put these requests in writing so both sides can judge the result against the same list.

- Name the domain and decision type, such as loan approvals or claims estimates, and keep to one of them at first. - State the condition types you want, and how many conditions a single decision may carry in the sample. - Ask for cases that end in hold, escalation and clean release, and say roughly what mix you want. - Request near-miss evidence that is attached but fails on date, entity, signature or threshold. - Ask for the schema file, a manifest, and expected end states with accepted reason codes for each task. - Say whether you need records only, a task pack, or a simulated environment with tools and reset. - Ask for separate training and held-out splits, divided by decision rather than by individual row. - List the licence questions to settle, including training use, evaluation use and publishing results.

Putting these requests in writing lets both sides judge the sample against the same list.

Audience:
Evaluation teams
Content type:
Checklist
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Approval conditions