What checks should I run before trusting an approval-authority task pack?
A checklist of verification steps for a sample task pack, covering date logic, limit arithmetic, gold outcomes, baselines and signs of repeated templates.
Run these checks on a sample before you build an evaluation on top of it, and keep the scripts so you can repeat them on later releases.
- **Single applicable authority:** each request resolves to one authority record at its decision date, or the ambiguity is documented. - **Independent arithmetic:** recompute limits, aggregates and delegation windows and compare them with gold outcomes. - **Boundary spread:** amounts fall below, at, just above and far above limits. - **Trivial baselines:** an always-approve and an always-escalate policy both score poorly. - **Template variety:** wording, evidence and routing history vary enough that a model cannot match on surface form. - **Policy agreement:** written rules and structured limits state the same thing. - **Practitioner read:** someone who has worked in the domain skims a sample of tasks for plausibility. - **Disclosure review:** scan for anything that looks like it identifies a real organisation or person.
Treat a failure on any item as a question for the supplier, not necessarily a reason to reject the pack. Some inconsistencies are deliberate edge cases, and the pack should say which.
Because the data is generated, passing these checks shows internal quality, not that the pack matches your production environment. Decide separately how well the domain and role structures fit your target use.
A short set of repeatable checks tells you whether a task pack can discriminate between careful and careless agents before you commit to wider use.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which failure modes should an approval agent evaluation include?
- How do I inspect the quality of generated authority rules and approval cases?
- How do environment reset, state diffs and grader hooks work for approval tasks?
- What success criteria should I use for an agent handling approvals?
- How does an evaluation catch an agent approving its own requester's credit adjustment?
- What checks should I run before accepting a generated approval sample?