What quality checks should buyers run before using a generated task pack?

Use these checks to establish whether a generated task pack can support interpretable training and evaluation results.

- Trace each expected action to supplied evidence. - Check linked records for contradictory obligation states. - Verify that necessary documents are actually accessible. - Confirm that unresolved cases permit abstention or escalation. - Keep related case families within one data split. - Remove answer labels from agent-visible metadata. - Review rare branches separately from routine reminders.

A reference outcome is useful only when the visible case supports it. If a label expects escalation based on a policy clause absent from the agent's inputs, the task measures access to hidden information. That defect should be corrected or the expected outcome revised.

Split integrity matters when several generated cases share the same underlying structure. Changing identifiers alone may leave the evaluation case recognisable from training. Ask for family identifiers that support grouping without exposing the intended answer.

MirrorFirm could provide an inspection manifest connecting cases to rules, generation families and intended branches. Buyers can use it to audit coverage and investigate failures while keeping evaluator-only material separate from the agent's working context.