What should I settle before requesting an escalation sample or pilot?
A checklist of decisions to make before requesting a sample or pilot, covering domain, form, use, case mix, difficulty, acceptance checks, review plan and format needs.
Before you request a sample or pilot, settle these points so the first delivery can be judged quickly.
- Domain and roles: which setting and how many approval levels you need - Form: dataset, task pack, simulated environment, or a mix - Use: training, evaluation, or both, and whether results will be published - Case mix: within-authority, over-limit, borderline, wrong-level and returned cases - Difficulty features: aggregated exposure, versioned limits, missing evidence - Acceptance checks: referential integrity, state transitions, gold outcomes, note quality - Review plan: who inspects grounding and disclosure risk, and what they will search for - Format needs: JSONL, Parquet, JSON Schema, tool definitions, reset method
Write the acceptance checks down before delivery, not after. It avoids arguing about quality once material has arrived. Ask for a data dictionary and a short account of how cases were constructed.
Keep the first scope narrow: one domain and one authority structure is easier to inspect than several, and what you learn tells you what to widen. Commercial terms, timing and pricing are agreed directly and are not described here.
Deciding these points first lets you judge the first delivery quickly and keep the initial scope narrow.
- Audience:
- Model labs
- Content type:
- Checklist
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What licensing questions should I raise for generated escalation material?
- What might a first escalation pilot show an agent team?
- What should my team prepare to start an escalation pilot?
- How should I scope a first escalation pilot with MirrorFirm?
- What should I ask for in an escalation sample or pilot?
- What licensing, limits and boundaries should a buyer ask about before a pilot?