How should a model lab define a useful first pilot?

Start with the capability the lab wants to understand. Evidence retrieval, policy application and interactive collection require different task designs. A pilot that tries to test all of them across unrelated processes can make a poor result difficult to interpret.

Choose a control family, such as independent approval of supplier changes, and define its operating boundary. Specify which process routes belong in scope, which records the model can access and whether it must gather evidence or only assess a prepared bundle. Include difficult cases that exercise the chosen capability without adding unrelated complexity.

The pilot should also state what it cannot establish. Success on supplier change reviews does not demonstrate competence across every audit process, and a generated task result does not establish that an organisation meets a regulatory requirement.

A useful first step is a short acceptance brief linking the intended model behaviour to the evidence needed to judge it. The buyer can use that brief to request a sample and decide whether to expand coverage, revise the task design or investigate a specific model weakness.