What evaluation hooks does an approval task pack need?
A task pack needs task specs, resettable snapshots, state-diff graders and trace checks, with result records that carry reason codes so failure types can be counted.
A task pack for this workflow needs graders that read state, not only final text. A workable structure has four parts.
**Task spec.** A JSON file per task with `task_id`, `start_snapshot`, `instruction`, `allowed_tools` and a `grader` block. The instruction might ask the model to process `REQ-2204` as the acting finance manager.
**Reset.** Each run starts from a named snapshot of the request, delegation and event tables. Resetting restores that snapshot, so repeated runs and different models are compared on the same state.
**State-diff grader.** After the run, the grader compares the tables with the expected end state. It checks the decision, the approver recorded, whether a second approval is pending, and that no unrelated rows changed.
**Trace checks.** The grader reads the tool-call log for required steps, such as looking up the delegation before recording an approval, and for prohibited ones, such as editing a limit table.
A result record could be:
`{"task_id":"T-0187","outcome":"fail","reason_codes":["approver_over_limit"],"checks":{"decision":false,"second_approver":true,"no_side_effects":true}}`
Reason codes let a team count failure types separately. Ask for the grader source, and for expected end states that can be reproduced from the policy and delegation tables, so you can audit the grader as well as the data.
Graders that read state and tool traces, with reason codes, let you measure which authority errors a model makes rather than a single pass rate.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which failure modes should an approval agent evaluation include?
- How do I inspect the quality of generated authority rules and approval cases?
- How do environment reset, state diffs and grader hooks work for approval tasks?
- What success criteria should I use for an agent handling approvals?
- How does an evaluation catch an agent approving its own requester's credit adjustment?
- What checks should I run before accepting a generated approval sample?