How do we check a sample before listing it for customers?
A three-pass review for internal quality, a list of common failure modes, a clear statement about what the checks do not show, and guidance on separating training from evaluation cases.
**How do we know a sample is good enough to list?**
Run three passes. First, load every table and check that keys resolve. Second, read a sample of cases against the policy text and see whether end states follow the rules. Third, count how many cases are exceptional and whether they differ from each other.
**What are the common failure modes in generated set-up data?**
- Orphaned records, such as a task with no account - Conditions on the approval that never reach the account - Timestamps that run backwards, such as activation before approval - Near-identical exception cases repeated with new identifiers - Policies that contradict each other across products
**Does passing these checks show the data is safe to license onward?**
No. These checks test internal quality. They do not prove anonymity, privacy compliance or regulatory approval, and generated data should not be described that way to your customers. Disclosure and grounding need their own review, and you should take your own legal advice for your markets.
**Can the data be used for evaluation and training?**
Yes, with separation. Hold out some cases as an evaluation set and keep them out of training sets, and record which split each customer received. Ask how a release can mark the split so it survives resale.
Internal quality checks are separate from disclosure review, and clean evaluation splits protect the meaning of scores your customers report.
- Audience:
- Licensed resellers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?