How can a buyer inspect the quality of generated sufficiency cases?
Describes four inspection areas, label derivation, consistency, coverage and grounding, and recommends baseline agents as a quick test of whether a set is too easy.
Quality in a generated sufficiency set comes down to whether the labels follow from the records, and whether the records behave like the real thing. A buyer can inspect four areas.
**Label derivation.** Recompute the expected decision from the requirement set and the links. If the gold label cannot be reproduced from the visible records by a careful reader, the case is flawed.
**Internal consistency.** Dates should order correctly, amounts should match across documents and a message should only mention documents that exist at that point in the status history.
**Coverage.** Count how cases spread across decisions, requirement sets and defect types. A set dominated by one decision lets a constant answer score well.
**Grounding and disclosure risk.** The data is generated, so it carries no guarantee about resemblance to real material. Check samples for names, identifiers or phrases that look as if they came from a real record, and ask how the generation process limits that. Also ask how the structure was informed by observation of real workflows, and what was deliberately not carried over.
Then run baselines: an agent that always proceeds, one that always asks, and a careful human reviewer. If the trivial agents score well, the set is too easy. If the reviewer often disagrees with the gold labels, the policy or labels need repair before the set is used for anything.
Recompute labels from the records, check coverage and run trivial baselines before trusting any generated set for scoring or training.
- Audience:
- Evaluation teams
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- How does a scored evaluation run expose a missed sufficiency check?
- Which sufficiency failure modes should I test before deploying an agent?
- How can I inspect the quality of generated sufficiency cases before using them?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?