What checks should we run on a sample before accepting it for evaluation use?

Run these checks on a small sample before you spend effort on a full run, and record the result of each so you can compare later releases.

- **Solve a subset by hand:** work out the outcome from policy and case, then compare with the key. Note every disagreement. - **Link integrity:** every foreign key resolves, and no document, party or prior case is orphaned. - **Label balance:** queues, statuses and outcomes are spread so that a constant answer scores poorly. - **Field-to-label dependence:** tabulate each incidental field, such as channel, against the expected queue to spot shortcuts. - **Trap coverage:** each failure mode you care about has several cases, and each is labelled in the key. - **Policy versions:** cases near a policy change date use the rule in force on the received date. - **Reset behaviour:** run the same task twice from reset and confirm identical starting state. - **Resemblance review:** scan names, addresses and identifiers for similarity to real people or organisations, and report anything found.

Treat any failed check as a question for the provider, not as a reason to drop the sample. Ask for a corrected version and note what changed.