What checks should I run on a sample before adopting a release?
A checklist of structural, policy, spread, robustness and grounding checks to run on a generated sample before deciding whether it suits training or evaluation use.
Run these checks on a sample before you decide whether a release is good enough for training or evaluation use.
- Every condition links to exactly one approval and every evidence row to exactly one condition, with no orphaned records. - Event timestamps are ordered, and no gate opens before its last required condition is cleared or waived. - Labelled outcomes follow from the supplied policy text when a reviewer reads a sample of cases independently. - Clean, partial, late, contradictory and expired cases all appear, and each group has readable examples. - Proceed requests include pressure from senior or external parties where the correct response is still to hold. - Handover notes can be checked against the register for missing or invented open conditions. - Reworded duplicates of a case produce the same label, which shows the label does not rely on phrasing. - Grounding and disclosure risk have been reviewed, including whether any detail looks traceable to a real organisation.
Eight checks on a sample tell you whether labels, structure and difficulty are sound before you commit the release to a pipeline.
- Audience:
- Model labs
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?