How can a buyer inspect the label quality of sufficiency data?
Label quality can be inspected by re-deriving outcomes from policy and documents, checking request wording against gaps, and reading a sample of borderline cases by hand.
The strongest check is re-derivation. Take a case, its documents and its policy rule, work out the outcome independently, and compare it with the label. A well-built dataset lets a reviewer do this using only the data supplied. If a label depends on knowledge that is not in the records, treat that as a defect.
A second check is internal consistency:
- every cited `rule_id` exists and applies to the case type - every gap points to a real document or marks it as absent - every request item corresponds to a gap, and every gap appears in the request - status history moves in an order the workflow permits
A third check is human reading. Pull a sample from the borderline region: documents that are almost recent enough, near-matching names, alternatives that qualify on one condition. These show where the generator's rules were loose.
Finally, look at distribution. Check that outcomes are not dominated by one class, that defect types vary, and that request wording is not a single template repeated.
For a reseller, these checks double as a quality statement you can give customers: here is how to verify it yourself. Do not state quality figures you have not measured.
The data is generated, so also check for grounding problems and for anything that resembles real records.
If outcomes can be re-derived from the supplied policy and documents, your customers can verify labels without relying on anyone's word.
- Audience:
- Licensed resellers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- How does a scored evaluation run expose a missed sufficiency check?
- Which sufficiency failure modes should I test before deploying an agent?
- How can I inspect the quality of generated sufficiency cases before using them?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?