What should happen when the buyer's referral policy changes during a pilot?

An evaluation lead reviews pilot case CASE-0924 for an insurance service team. Under policy POL-0107, a specified product routes to the general support queue. A revised policy routes that product to a dedicated specialist queue. The customer's circumstances and account records remain unchanged.

The lead requests a revised generated case linked to the new policy and updates the expected routing action. The previous case remains associated with its original policy. The pilot manifest changes to identify which version each evaluation run uses.

Without this distinction, an agent following the earlier policy could appear wrong against a silently updated answer key. Conversely, an agent using the new route could receive credit in a task whose visible policy still instructs something else.

This scenario teaches a practical scoping lesson: ask how policy changes will be represented before using the pilot for comparisons. Separate corrections to defective material from deliberate changes in business rules. The buyer needs enough version information to explain whether a changed score reflects agent behaviour, revised policy or a changed evaluation specification.