What checks should I run before accepting an intake sample?
A checklist of file-level and run-level checks, covering keys, transitions, timestamps, rule coverage, reference solutions, determinism, protected fields and disclosure review.
Before accepting a sample, run these checks on the files and on a small number of agent runs, and record the result for each.
- Every foreign key resolves: documents to cases, cases to parties, cases to queues and owners - Status values and transitions in the data match the declared transition table, with no skipped states - Timestamps run forward within each case, and scripted events arrive after the actions that trigger them - Each policy rule is exercised by at least one case, and each exception case maps to a rule - A reference solution passes every task, and a careless always-route baseline fails the tasks with gaps - Resetting twice from the same snapshot and seed gives matching content hashes - Protected fields are listed per task and the grader fails any run that alters them - A sample of names, identifiers and free text is reviewed for resemblance to real parties
These checks are scriptable and repeatable, so you can compare candidate releases on the same footing.
- Audience:
- Environment builders
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What should you check when you receive an intake sample?
- What evaluation hooks should a simulated intake environment expose?
- What failure modes should an intake evaluation measure?
- How can a buyer judge the quality of generated intake data?
- How does a graded run expose a skipped completeness check?
- How can a buyer inspect the quality of a generated intake environment?