What failure modes should we test for before relicensing an intake dataset?

**What failure modes should we test for before relicensing an intake dataset?**

- **Label leakage.** The queue name or outcome appears in a free-text field or file name, so a model can read the answer. - **Rule drift.** Records cite a rule version that does not match the outcome. - **Over-clean data.** Few blanks, duplicates or conflicts, so models struggle on first contact with real submissions. - **Split leakage.** Events from one case appear in both training and test sets. - **Template echo.** Free text repeats a small set of phrasings, which inflates apparent performance. - **Unreasoned labels.** Outcomes without reason codes, so evaluators cannot judge disagreements. - **Disclosure risk.** Distinctive strings or combinations that could point to a real person or business.

**How do we test for template echo?**

Cluster the free text and look at near-duplicates. Count distinct openings and phrasings per domain, and compare them with the number of cases.

**How do we test for label leakage?**

Train a simple model on the free text alone and see whether it predicts the queue much better than a chance baseline. Treat a strong result as a flag to investigate.

**Who decides what is acceptable?**

You do, together with your customers. Agree thresholds before listing, and record them against the release version.