Which checks should I run on a sample before committing to wider use?
A checklist of integrity, consistency, ground-truth and reset checks that show whether a sample is sound enough to build evaluation on.
Run these checks on a sample before you commit to wider use of a release, and record the result of each so the review can be repeated.
- Every condition links to an existing decision, and every evidence item links to an existing condition. - Timestamps are ordered sensibly: evidence follows the condition opening, and decisions precede their amendments. - Policies do not contradict each other, and each waiver cites a policy that actually permits it. - Each task has an expected action, a forbidden action and a reason code that a practitioner would accept. - Blocked, cleared and ambiguous cases are all present, with no single text cue that cleanly separates them. - Free-text notes agree with the structured state, except where a task deliberately plants a mismatch. - Resetting the environment returns it to the identical starting state on repeated runs of the same task. - Nothing in the content looks lifted from a source you recognise, and any doubt is raised with MirrorFirm.
A short, repeatable set of acceptance checks catches broken links, contradictory policies and leaky tasks while changes are still cheap.
- Audience:
- Agent developers
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?
- How can I inspect the quality of generated conditional approval material?