What checks should I run on a pilot sample before extending it?
A checklist of eight inspection steps covering rule tracing, status validity, free-text review, case mix, reviewer agreement, date consistency, baseline failures and reset repeatability.
Run these checks on a pilot sample before deciding whether to extend it.
- Trace ten decisions back to numbered rules and confirm each rule really supports the stated outcome. - Confirm every status in the cases belongs to the closed list and every recorded transition is allowed. - Read the free text for names, addresses or phrasing that looks copied from a real source, and report anything suspect. - Check that the mix includes clean cases, incomplete cases, duplicates and cases that require escalation. - Have two reviewers label a set of tasks independently, then compare their answers with the reference outcomes. - Verify that dates, thresholds and effective periods agree across records and policies. - Run a baseline model and read its failures to find ambiguous tasks and weak references. - Reset the environment twice from the same seed and confirm the starting state is identical each time.
These checks find ambiguous tasks, inconsistent rules and disclosure concerns early, while the sample is still small enough to read in full.
- Audience:
- Model labs
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What should you check when you receive an intake sample?
- What evaluation hooks should a simulated intake environment expose?
- What failure modes should an intake evaluation measure?
- How can a buyer judge the quality of generated intake data?
- How does a graded run expose a skipped completeness check?
- How can a buyer inspect the quality of a generated intake environment?