What does a first pilot review look like for a referral-processing agent team?

An agent team building a referral-processing assistant for a healthcare administration provider asks for a pilot sample. Their evaluation lead writes a one-page request: referral approvals with conditions, three condition types (a prior-authorisation document, a clinical review sign-off and a funding check), four exception types and a JSONL format with stable identifiers such as REF-2290.

The sample arrives as a small set of decision, condition and evidence files plus a task file with expected and forbidden actions. The evaluation lead and a practitioner from the administration team read half of the tasks by hand. They find that one policy lets a clinical reviewer waive the funding check, while another says only the finance officer can. The conflict is logged and sent back as a scoping note.

The team then runs two baselines. A status-only agent proceeds on every approved referral and fails the blocked tasks. A wait-always agent fails every cleared one. Their own prototype lands between the two, and its failures concentrate on stale evidence.

The outcome is a revised request: the policy conflict resolved, more stale-evidence cases, and a second release in Parquet for their analytics team. The decision to widen rests on the baseline separation and the failure breakdown.

**What it teaches:** a pilot is a way to test the data as much as the agent. Hand review, baselines and a written scoping note turn a small sample into a clear next request.