How can I inspect the quality of a generated approval-authority dataset?
Describes checks an evaluation team can run on a generated sample, covering internal consistency, boundary coverage, outcome validity and signs of weak grounding.
Start with mechanical checks, because they are cheap and catch the most damaging faults.
**Consistency.** Every request should resolve to exactly one applicable authority record at its decision date. Recompute limits, aggregates and delegation windows independently and compare with the gold outcome. Any task where your computation disagrees is either a data fault or an ambiguity to be documented.
**Boundary coverage.** Plot amounts against limits. A useful pack has items well under, exactly at, just over and far over each limit, not only clear-cut cases.
**Outcome validity.** Check that permitted outcomes include sensible alternatives, such as escalating to either eligible approver, and that no gold answer relies on a rule absent from the policy text.
**Distribution.** Look at the mix of condition tags, roles and categories. Heavy repetition of one pattern means a model can learn the template instead of the reasoning.
Then read a sample by hand. Ask whether the routing history, comments and evidence read like the work of the named roles, and whether an experienced practitioner would find the limits plausible.
Finally, treat grounding and disclosure as open questions. The data is generated, informed by observation of how businesses operate. Ask what the generation was based on, and look for anything that appears to reproduce a specific organisation's details.
Most quality problems in authority data can be found by mechanical checks on dates, limits and gold outcomes, plus a manual read for realism.
- Audience:
- Evaluation teams
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which failure modes should an approval agent evaluation include?
- How do I inspect the quality of generated authority rules and approval cases?
- How do environment reset, state diffs and grader hooks work for approval tasks?
- What success criteria should I use for an agent handling approvals?
- How does an evaluation catch an agent approving its own requester's credit adjustment?
- What checks should I run before accepting a generated approval sample?