What checks should we run on a sample before accepting it for evaluation or training?

Run these checks on a sample before you accept a sufficiency set for evaluation or training use.

- Recompute gold decisions from the requirement set and links, and confirm they match - Check that dates, amounts and names agree across documents within each case - Confirm the spread of decisions, so that no single constant answer scores well - Run always-proceed and always-ask baselines and review how they score - Have a domain reviewer read a handful of cases and flag any that feel implausible - Inspect text for strings that resemble real identifiers, people or organisations - Verify that gold labels are held apart from files the agent can read - Re-run generation with the same settings and confirm outputs are stable, if you expect reproducibility

Record what you find, including cases you would reject, and send that list back with the sample. Specific rejected cases are more useful for adjusting a release than general comments about realism. Remember that the data is generated, so a clean pass on these checks shows fitness for your test design, not that the material is free of disclosure risk or approved for any regulatory purpose.