Skip to content
MirrorFirm

How can a buyer inspect the quality of generated sufficiency cases?

Describes four inspection areas, label derivation, consistency, coverage and grounding, and recommends baseline agents as a quick test of whether a set is too easy.

Quality in a generated sufficiency set comes down to whether the labels follow from the records, and whether the records behave like the real thing. A buyer can inspect four areas.

**Label derivation.** Recompute the expected decision from the requirement set and the links. If the gold label cannot be reproduced from the visible records by a careful reader, the case is flawed.

**Internal consistency.** Dates should order correctly, amounts should match across documents and a message should only mention documents that exist at that point in the status history.

**Coverage.** Count how cases spread across decisions, requirement sets and defect types. A set dominated by one decision lets a constant answer score well.

**Grounding and disclosure risk.** The data is generated, so it carries no guarantee about resemblance to real material. Check samples for names, identifiers or phrases that look as if they came from a real record, and ask how the generation process limits that. Also ask how the structure was informed by observation of real workflows, and what was deliberately not carried over.

Then run baselines: an agent that always proceeds, one that always asks, and a careful human reviewer. If the trivial agents score well, the set is too easy. If the reviewer often disagrees with the gold labels, the policy or labels need repair before the set is used for anything.

Recompute labels from the records, check coverage and run trivial baselines before trusting any generated set for scoring or training.

Audience:
Evaluation teams
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Document sufficiency