What should an evaluation team ask for in a first sample or pilot?
A checklist for requesting a first sample, covering case mix, expected states, policy text, format and the review you will do on arrival.
Use this list to specify a first sample so you can judge fit quickly.
- One domain and one target record type, stated plainly. - A mix of clean controls and named exception cases, with the count of each you want. - An expected end state for every case, including hold and escalate outcomes. - The governing policy text and its structured form, so you can check they agree. - A file format and schema description that match your harness. - Seed and reset details so runs can be repeated. - A note on which fields are templated and which vary, so you can judge variety. - A named practitioner on your side to review a sample of cases for realism.
On arrival, run the structural checks first, then grade a small number of runs by hand before trusting automated scores.
The material is generated and synthetic, so include in your plan an inspection of grounding and disclosure risk. Agree prices, timelines and licence terms directly with MirrorFirm, as none are set here.
A well-specified request gets a sample you can judge in days of review rather than weeks of back and forth.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot look like for a client and matter opening agent?
- What licence and usage limits should we ask about before starting?
- How should we scope a first pilot for set-up agents?
- What should we ask for in a sample or pilot release for set-up?
- How do we wire a pilot release into our agent harness on day one?
- How should delivery formats, versioning and data splits be agreed for a pilot?