What checks should I run on a sample before accepting it?
A practical list of structural, policy and plausibility checks that a technical reviewer can run on a generated sample of conditional-approval records and tasks.
Run these checks on any sample, and record failures by identifier so they can be traced and fixed.
- Every foreign key resolves, with no orphan conditions, evidence items or actions. - Status transitions match the allowed list, and no record skips a required step. - Timestamps are ordered correctly, including evidence dates against condition dates. - Policy text and records agree, especially on waiver authority and evidence age. - Gated actions never appear as completed while a blocking condition is open, except in seeded violation cases that are labelled as such. - Task prompts have an expected outcome, and a second reviewer reaches the same answer. - Exception cases are present and differ from each other in cause and handling. - Text fields are checked for anything that looks copied from a real source.
If several checks fail in the same place, raise it as one pattern rather than many separate defects. That helps decide whether to change the schema, the policy text or the generation settings.
These checks turn a vague sense of quality into specific pass or fail results you can share with MirrorFirm and your own team.
- Audience:
- Environment builders
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?