Skip to content
MirrorFirm

What evaluation hooks does an approval task pack need?

A task pack needs task specs, resettable snapshots, state-diff graders and trace checks, with result records that carry reason codes so failure types can be counted.

A task pack for this workflow needs graders that read state, not only final text. A workable structure has four parts.

**Task spec.** A JSON file per task with `task_id`, `start_snapshot`, `instruction`, `allowed_tools` and a `grader` block. The instruction might ask the model to process `REQ-2204` as the acting finance manager.

**Reset.** Each run starts from a named snapshot of the request, delegation and event tables. Resetting restores that snapshot, so repeated runs and different models are compared on the same state.

**State-diff grader.** After the run, the grader compares the tables with the expected end state. It checks the decision, the approver recorded, whether a second approval is pending, and that no unrelated rows changed.

**Trace checks.** The grader reads the tool-call log for required steps, such as looking up the delegation before recording an approval, and for prohibited ones, such as editing a limit table.

A result record could be:

`{"task_id":"T-0187","outcome":"fail","reason_codes":["approver_over_limit"],"checks":{"decision":false,"second_approver":true,"no_side_effects":true}}`

Reason codes let a team count failure types separately. Ask for the grader source, and for expected end states that can be reproduced from the policy and delegation tables, so you can audit the grader as well as the data.

Graders that read state and tool traces, with reason codes, let you measure which authority errors a model makes rather than a single pass rate.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Delegated approval