What checks should I run before accepting a generated approval sample?
A repeatable set of structural, boundary, consistency and reproducibility checks to run on a sample before building graders or training runs on it.
Run these checks on a sample before building graders or training runs on it, and record the results so they can be repeated on later releases.
- Every foreign key resolves, including decision to request, rule and delegation, with no orphan identifiers. - Authority at decision time can be recomputed from the tables alone, and your result matches the release's validity labels. - Boundary amounts, such as exactly equal to a limit, are handled the same way throughout. - Delegation windows do not overlap in ways the policy forbids, and expired ones are not used except in labelled traps. - Amounts, currencies and timestamps are well formed, with no negative approvals or decisions dated before the request existed. - Narrative text agrees with structured fields, so a note saying urgent medium risk does not sit on a low-risk row. - Invalid cases are distributed across failure types rather than concentrated in one. - A re-run with the same seed reproduces the same files, so the sample is stable.
If more than a few checks fail, ask for an explanation before widening scope. Some failures are intended, such as seeded traps, so confirm which are labelled.
Also look for generic sameness: repeated role names, copy-pasted narratives or identical amounts suggest the variety you need for training is thin.
Treat this as an inspection of generated material, with its own grounding and disclosure risks, rather than as proof of fitness for any production use.
Record the results of these checks so they can be repeated on later releases, and treat a cluster of failures as a reason to pause scope.
- Audience:
- Environment builders
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which failure modes should an approval agent evaluation include?
- How do I inspect the quality of generated authority rules and approval cases?
- How do environment reset, state diffs and grader hooks work for approval tasks?
- What success criteria should I use for an agent handling approvals?
- How does an evaluation catch an agent approving its own requester's credit adjustment?
- How do I check that generated approvals are consistent with their rules?