How do we test generated case records for disclosure risk?
Check for near-copies against material you hold, unusual rare combinations, odd detail in free text and identifier formats, and ask how the release was generated.
**No test can prove the absence of risk, but several checks are worth running.**
**Near-copy search.** If you hold any real or third-party records in the same domain, compare generated text and field combinations against them. Close matches in rationales or unusual amounts deserve a closer look.
**Rare combinations.** List attribute combinations that occur only once, such as an uncommon product, role and amount together. Rare combinations are where a generated record could resemble something specific.
**Free-text review.** Read a sample of handover notes and rationales for oddly specific detail, such as unusual place names, dates or phrasing that seems lifted from a source.
**Identifier formats.** Confirm identifiers like `APP-2187` follow the generated pattern and do not resemble real account or tax number formats.
**Generation notes.** Ask how the release was produced, what grounding it reflects, and what was done to limit leakage.
**What to say in your listing.** State that the data is generated and that buyers should run their own disclosure checks. Do not describe the release as anonymous, privacy-compliant or approved by any regulator.
If a check raises a concern, report the specific records back and ask for a corrected release before you list it.
Disclosure risk can be examined with concrete tests, though passing them lowers concern rather than removing it, so keep your own review.
- Audience:
- Licensed resellers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- Which failure modes should an escalation evaluation cover?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?