How should we scope a useful first pilot?

Start by naming the decision your evaluation needs to assess. A useful scope is whether an agent can determine that a supplier setup request has sufficient evidence for internal review. That is more actionable than asking for a broad collection of audit documents.

Define the process boundary: where the request begins, which control is being reviewed and what constitutes completion. Specify what the agent may see and do, which exceptions belong in scope and which decisions remain with a human reviewer. State whether the pilot is intended for training, evaluation or comparison of alternative agent designs.

Ask for a pilot manifest that lists the proposed artefacts, task instructions, expected outputs and acceptance criteria. Include an explicit list of excluded behaviours, such as independently judging the adequacy of the organisation’s control design.

A useful first step is to supply a written review brief and your evaluation harness requirements. The proposed generated sample can then be assessed against a concrete need. Broader coverage should follow evidence that the initial task exposes meaningful differences in agent behaviour, rather than merely demonstrating that the files can be loaded.