How should we test generated sufficiency data for grounding and disclosure risk?
Test by searching for near-copies of known sources, checking identifiers and free text for unusual specificity, and asking the supplier how generation was grounded.
**Does generated mean safe to pass on?**
No. Generated data can still resemble real material or reproduce detail from its sources. Do not tell customers it is anonymous, compliant or approved. Inspect it.
**What can we check?**
- **Identifier patterns.** Look for names, addresses, account numbers or reference formats that look unusually specific or repeat across cases. - **Free-text fields.** Read request messages and rationales for phrases that read like copied correspondence. - **Near-duplicates.** Compare documents within the release and against any material you hold for similarity. - **Rare combinations.** A very unusual mix of attributes in a single case can point back to a source. - **Outliers.** Check the longest and most specific records first.
**What should we ask the supplier?**
Ask how the content was generated, what the generation was informed by, what filtering was applied for identifiers, and what limits the supplier states on use. Ask for these in writing and keep them with the release.
**Can this be automated?**
Part of it can, such as pattern scans and similarity checks. Reading a sample by hand still finds things the scans miss.
Decide before onward licensing who in your organisation signs off on this review, and keep a record of what was checked.
Disclosure review is your responsibility to repeat before onward licensing, so build it into your intake checks and not just rely on the brief.
- Audience:
- Licensed resellers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- How does a scored evaluation run expose a missed sufficiency check?
- Which sufficiency failure modes should I test before deploying an agent?
- How can I inspect the quality of generated sufficiency cases before using them?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?