What should you check when you receive an intake sample?
A checklist of consistency, policy, exception-mix, realism, disclosure and reset checks a buyer can run on a first generated sample before committing to anything larger.
Run these checks on a sample before deciding whether the material suits training or evaluation.
- Every foreign key resolves: cases to parties, documents to cases, amendments to earlier cases. - Status histories follow allowed transitions and timestamps never run backwards. - Expected outcomes can be reproduced by applying the written policies to the case. - The mix of clean, incomplete, duplicate and conflicting cases matches the stated design. - Documents and notes read like staff work, with shorthand, gaps and inconsistent formatting where intended. - No name, identifier or phrase resembles a real person or organisation; flag any that do. - Tasks include forbidden actions as well as target outcomes, so shortcuts can fail. - Two reruns of the same task from reset produce the same starting state.
Keep a record of which checks passed, and note the release identifier alongside the results. The same list can be reused when a larger release arrives. Failures at this stage are cheap to raise; failures found after training are not.
Cheap checks at sample stage catch specification problems before they surface after training or evaluation work has begun.
- Audience:
- Agent developers
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a simulated intake environment expose?
- What failure modes should an intake evaluation measure?
- How can a buyer judge the quality of generated intake data?
- How does a graded run expose a skipped completeness check?
- How can a buyer inspect the quality of a generated intake environment?
- How can a run that reaches the right queue still be a failure?