How can I inspect the quality of generated sufficiency data before using it?
Quality can be inspected through policy consistency, label correctness, defect coverage, realism review by domain staff, and checks for grounding and disclosure risk.
Inspect a sample on five fronts before committing to a larger release.
**Internal consistency.** Recompute the gold decision from the policy and the document labels. If your own script cannot reproduce the gold gaps, the labels or the policy are wrong. This check is cheap and should be automated.
**Label correctness.** Have a domain reviewer read a subset of cases and write their own decision before seeing the gold. Disagreements point to ambiguous policies or mislabelled defects.
**Coverage.** Count cases by defect type, outcome and domain. Look for gaps such as no escalations, or conflicts that always involve the same document type.
**Realism.** Ask someone who has done this work whether the documents, request wording and replies read like the real thing. Note where the generated version is too tidy.
**Grounding and disclosure.** The material is generated, not real records, but it is informed by observation of real operations. Check that no document reads as a copy of an identifiable record, and ask how grounding is documented.
None of these checks proves the data will improve a particular model. They tell you whether the data is internally sound and plausible. Improvement is a separate question, answered by an experiment on your own held-out tasks, with and without the data.
A defined inspection routine lets a buyer accept or reject a sample on evidence instead of on impressions.
- Audience:
- Model labs
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- How does a scored evaluation run expose a missed sufficiency check?
- Which sufficiency failure modes should I test before deploying an agent?
- How can I inspect the quality of generated sufficiency cases before using them?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?