What should a buyer request in a reviewable sample?

Request a sample that lets evaluators inspect an entire task from initial instructions through accepted outcomes.

- Include the exact agent-facing prompt and records. - Supply the governing policy with clause references. - State the allowed tools and completion conditions. - Provide accepted outcomes and concise supporting rationales. - Identify intentional ambiguity and missing information. - Include a case requiring clarification or deferral. - Document known defects and excluded behaviours. - State the sample's permitted uses and handling conditions.

Review the sample using the same interface planned for the eventual benchmark. A task that is understandable in a presentation may become ambiguous when its records are retrieved through separate tools. Conversely, a sample may expose too much evaluator information when flattened into a single prompt.

MirrorFirm could use the review findings to refine the generated pack's scope. Record disagreements as concrete issues: a missing policy precedence rule, an inaccessible record or an accepted answer that conflicts with the state transition. Avoid reducing the review to whether the narrative feels realistic.

The useful first step is agreement on what counts as a valid task. Once that is clear, additional cases can extend coverage without multiplying unresolved specification problems.