What should a lab specify to keep a pilot evaluation useful after sample inspection?

Use this checklist to agree how sample inspection and subsequent evaluation will remain separate.

- Name the capability the holdout should measure. - Identify which examples reviewers may inspect fully. - Specify which policy families cross split boundaries. - Keep answer keys outside model-facing retrieval stores. - Record who receives protected tasks and annotations. - Define what exposure makes a task unsuitable. - Request fresh variants when protected answers are exposed.

The split should match the intended claim. If the lab wants to test generalisation to an unfamiliar policy structure, changing record identifiers while retaining the same decision logic is insufficient. If it wants to test execution under a familiar rule, reusing the policy family may be appropriate while holding out work records and event sequences.

Sample review can still be thorough. Reviewers can inspect a designated development set, challenge its expected outcomes and request corrections without opening the protected evaluation set. The brief should explain which similarities between sets are deliberate.

A useful first step is to describe the generalisation question before asking for a sample. MirrorFirm could then scope generated partitions around that question rather than treating a random record split as automatically suitable.