What quality checks should precede evaluation runs?

Apply these checks to the generated package before admitting its tasks into an evaluation suite.

- Resolve every referenced record and policy clause. - Check event ordering against permitted lifecycle transitions. - Match structured rule differences to the policy text. - Confirm each expected action has supporting evidence. - Review role permissions against expected tool use. - Inspect grounding documentation and disclosure risk. - Have a second reviewer challenge disputed outcomes.

A task can look plausible while containing a decisive contradiction. A case history might record approval before the required evidence existed, while the evaluator assumes the evidence was available. Unless that inconsistency is deliberately part of the task, it prevents a clean interpretation of the agent's result.

Record review findings at task level so disputed cases can be withheld without discarding unrelated material. Keep construction defects separate from intentionally difficult business conditions, such as an incomplete notice or conflicting operational instructions.

Generated output still needs scrutiny of how its grounding is represented and what details it contains. That review complements schema validation; neither substitutes for the other. Admit a task when its intended challenge and acceptable outcomes are clear enough for independent evaluators to assess consistently.