What checks should an evaluation team run on a first sample?
A checklist of eight inspection checks for a first sample, covering link integrity, timing, rule fidelity, waiver authority, outcome mix, label review and disclosure risk.
Run these checks on a first sample before you spend time building tasks on top of it.
- Follow every condition to its decision, and confirm no condition points at a missing or mismatched decision record. - Confirm evidence is received after its condition exists, and that each review follows the evidence it assesses. - Read a sample of findings against rule text, checking that threshold, date and entity tests are applied correctly. - Look for evidence that is attached but should fail, and confirm the expected status reflects that failure. - Check that waivers and variations carry an authorising role that the stated policy actually permits. - Tally how many cases end in hold, escalation or clean release, and confirm the mix suits your evaluation goals. - Read several expected end states yourself and mark any where a careful reader of the policy would disagree. - Scan text fields for content that looks lifted from a real source or names a real party, and report anything found.
Run these checks before building tasks on a sample, so defects in the data do not become defects in your scores.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?