How should an evaluation team scope its first policy-change pilot?

Begin with a single business decision that changes under a clearly specified policy revision. Define the agent's role, the records it can inspect and the actions it can take. This gives the pilot a stable centre even if the surrounding workflow contains several handovers.

Choose a boundary that preserves meaningful reasoning. A retail refund approval change can include eligibility evidence and reviewer routing while excluding stock valuation and payment settlement. Those exclusions should be explicit so an agent is not expected to resolve consequences the environment cannot represent.

Ask for a sample that includes a task statement, visible records, evaluator rationale and a completed reference trace. Review whether the task exercises the intended capability and whether the trace demonstrates an acceptable path rather than the only permissible sequence.

Agree on what would justify expansion: perhaps the task distinguishes applicability mistakes from tool-use mistakes, or reviewers can consistently assess alternative valid outcomes. Add further rules, roles or systems only when they answer a clear evaluation question. The first step is to write a bounded decision brief that a sample can be assessed against.