What should we check in a sample before accepting it?

Run these checks on a first sample, before spending time on agent runs, because they catch problems in the data itself.

- Rebuild the answer key from the requirement table with your own script and confirm it matches the supplied key. - Count cases by outcome class and defect type, and confirm the mix suits the boundary you want to test. - Read a spread of documents by hand for templated phrasing, obvious defects and implausible requirements. - Confirm a domain reviewer finds the ideal request wording plausible for the business in question. - Reset the environment twice from one seed and diff the full state to confirm repeatability. - Send malformed tool arguments and confirm the environment returns clear errors without corrupting state. - Search generated text for passages that look copied from public or real sources, and for identifiers resembling real ones. - Check that agent-visible files contain no defect tags or key fields that would leak the answer.