What checks should a buyer run on a sample before adopting a release?
A checklist of seven checks covering referential integrity, status history, derived fields, answerability, reviewer agreement, baselines and disclosure review.
Run these checks on a sample before you commit to a larger release; each can be scripted or reviewed by hand.
- **Referential integrity.** Every offer, facility, party link and security record points to an existing parent, with no orphans. - **Status history.** Only legal transitions appear, and each hold or release carries a reason code and a role. - **Derived fields.** Schedules, fees and limits recompute correctly from the stated terms. - **Answerability.** Each defect in an exception case can be found using only the records and policy supplied. - **Single defensible outcome.** Two independent reviewers reach the same expected result when reading the same cases. - **Baselines.** A scripted agent and a weak model score low on exception cases, and a careful run scores higher. - **Disclosure review.** Names, entities and document wording are checked for resemblance to real organisations.
These checks can be scripted or reviewed by hand on a small sample, and they expose most structural problems before a larger release is licensed.
- Audience:
- Model labs
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- How do I know whether my success criteria for booking tasks are strict enough?