How should an evaluation team choose a useful first pilot?

Start by naming the decision your current evaluation cannot assess. A useful pilot question might be whether an agent can distinguish a change it may complete from one requiring a specialist review. That question is more actionable than asking for broad coverage of customer changes.

Define the entry condition, the records visible to the agent, the actions it may take and the point at which the episode ends. Also specify the behaviours outside scope. A pilot can test referral decisions without modelling the specialist's subsequent investigation, provided the acceptance criteria stop at a well-formed referral.

Ask for a sample that exercises both sides of the selected boundary and includes the policy evidence needed to justify each outcome. Review it with whoever owns the evaluation methodology and whoever can assess the business logic. Resolve disagreements about the expected answer before running a model comparison.

MirrorFirm could scope generated material around this bounded question. The first deliverable to agree is an acceptance specification describing what the pilot can establish and what remains untested. Expansion can follow identified gaps rather than an assumption that more records automatically produce a better benchmark.