How can I inspect the quality of generated sufficiency cases before using them?
Quality inspection combines automated consistency checks across linked records with a domain reviewer's reading of borderline cases, plus a review of grounding and disclosure risk.
Inspect a sample in layers, starting with checks a script can run and ending with ones that need a person who knows the process.
**Mechanical integrity.** Every gap should reference an existing requirement and document. Every `sufficient` case should satisfy all of its requirements under the stated policy version. Every `insufficient` case should have at least one gap that actually fails. Dates should be consistent: a document cannot be received before it is issued.
**Label logic.** Recompute the expected outcome from requirements and documents with your own script and compare it to the supplied label. Disagreements point to label errors or to rules that the data does not state.
**Variety.** Check that case types, document faults and reply patterns are spread, and look for near-duplicate cases that would inflate a test set.
**Domain review.** Ask someone who has handled these cases to read the borderline ones. Would they accept the substitute? Is the request wording something a handler would send?
**Grounding and disclosure.** The output is generated, so ask what each element is based on, and check samples for details that look too specific to a real source.
Record what you find. A short defect log from a pilot is the most useful input to the next release.
Run mechanical integrity checks first, then spend reviewer time on the borderline cases where labels are most likely to be debatable.
- Audience:
- Agent developers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- How does a scored evaluation run expose a missed sufficiency check?
- Which sufficiency failure modes should I test before deploying an agent?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?
- How can a buyer inspect the quality of a sufficiency dataset or environment?