How can sample review reveal that a pilot needs a different task boundary?

A model lab requests referral tasks for a logistics billing assistant. The initial scope covers a customer describing difficulty paying freight invoices and the assistant routing the contact to a specialist team. During sample review, an evaluator examines contact CON-2187, invoice INV-0412 and account ACC-0634.

The evaluator identifies the real target capability: the assistant must determine which trading account the contact can discuss before preparing the referral. The proposed sample assumes that relationship is already resolved, so it does not exercise the intended failure mode.

The pilot specification changes from conversation-level routing to account-scoped intake. The revised generated material adds an authority lookup, separate trading accounts and a policy for unresolved representation. The expected endpoint remains a referral, but the model must now establish its permitted scope first.

The consequence is a more relevant evaluation without expanding into specialist assistance decisions. This teaches buyers to review what the sample assumes, not just what it contains. A useful pilot revision removes an assumption that hides the capability the lab needs to measure.