Skip to content
MirrorFirm

How can I inspect the quality of a generated approval-authority dataset?

Describes checks an evaluation team can run on a generated sample, covering internal consistency, boundary coverage, outcome validity and signs of weak grounding.

Start with mechanical checks, because they are cheap and catch the most damaging faults.

**Consistency.** Every request should resolve to exactly one applicable authority record at its decision date. Recompute limits, aggregates and delegation windows independently and compare with the gold outcome. Any task where your computation disagrees is either a data fault or an ambiguity to be documented.

**Boundary coverage.** Plot amounts against limits. A useful pack has items well under, exactly at, just over and far over each limit, not only clear-cut cases.

**Outcome validity.** Check that permitted outcomes include sensible alternatives, such as escalating to either eligible approver, and that no gold answer relies on a rule absent from the policy text.

**Distribution.** Look at the mix of condition tags, roles and categories. Heavy repetition of one pattern means a model can learn the template instead of the reasoning.

Then read a sample by hand. Ask whether the routing history, comments and evidence read like the work of the named roles, and whether an experienced practitioner would find the limits plausible.

Finally, treat grounding and disclosure as open questions. The data is generated, informed by observation of how businesses operate. Ask what the generation was based on, and look for anything that appears to reproduce a specific organisation's details.

Most quality problems in authority data can be found by mechanical checks on dates, limits and gold outcomes, plus a manual read for realism.

Audience:
Evaluation teams
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Delegated approval