Skip to content
MirrorFirm

How can I inspect the quality of generated sufficiency data before using it?

Quality can be inspected through policy consistency, label correctness, defect coverage, realism review by domain staff, and checks for grounding and disclosure risk.

Inspect a sample on five fronts before committing to a larger release.

**Internal consistency.** Recompute the gold decision from the policy and the document labels. If your own script cannot reproduce the gold gaps, the labels or the policy are wrong. This check is cheap and should be automated.

**Label correctness.** Have a domain reviewer read a subset of cases and write their own decision before seeing the gold. Disagreements point to ambiguous policies or mislabelled defects.

**Coverage.** Count cases by defect type, outcome and domain. Look for gaps such as no escalations, or conflicts that always involve the same document type.

**Realism.** Ask someone who has done this work whether the documents, request wording and replies read like the real thing. Note where the generated version is too tidy.

**Grounding and disclosure.** The material is generated, not real records, but it is informed by observation of real operations. Check that no document reads as a copy of an identifiable record, and ask how grounding is documented.

None of these checks proves the data will improve a particular model. They tell you whether the data is internally sound and plausible. Improvement is a separate question, answered by an experiment on your own held-out tasks, with and without the data.

A defined inspection routine lets a buyer accept or reject a sample on evidence instead of on impressions.

Audience:
Model labs
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Document sufficiency