How should an evaluation team scope a pilot for conditional approval tasks?

A useful first scope is narrow. Pick one domain, one type of decision and a small set of condition types, then widen once the sample shows the structure works for your harness.

Decide what you need to test. Some teams want tasks that check whether an agent respects blocked steps. Others want data for tuning on evidence review. These lead to different mixes: the first needs more near-miss evidence and held-out cases, the second needs more review records with clear findings.

Specify the dimensions to scope in a pilot:

- the domain and decision type, such as loan approvals, claims estimates or connection applications - the condition types and how many sit on a single decision - the share of cases that should end in a hold or escalation - whether waivers, variations and lapses are included - the output form: records only, a task pack with expected end states, or a simulated environment with tools

Limits apply. The output is generated, so it does not replace testing against your own systems, and it is not evidence of regulatory compliance. Grounding and disclosure risk should be inspected in the sample. Licence terms, scale and delivery arrangements are agreed in the engagement, so raise them early rather than assuming defaults.

A sensible first step is a small sample and a written list of the checks you will run on it.