What evaluation hooks and grader outputs can a task pack provide?
A task pack could expose tool actions, a reset command, a final state diff and per-check results, so buyers can run repeatable evaluations in their own harness.
A task pack for this workflow can define a small set of hooks.
**Tool actions.** Typical actions include `get_case`, `lookup_authority`, `get_policy`, `escalate_case`, `record_decision`, `add_condition` and `request_document`. Each has a typed input and a result, so an agent's calls can be logged and replayed.
**Reset.** A `reset(task_id, seed)` call restores the environment to the snapshot named by `initial_state_ref`. With the same seed, a run starts from identical state, which makes comparisons between model versions meaningful.
**Final state diff.** After a run, the grader can output the difference between actual and expected state, limited to the tables the task touches.
**Check results.** A structured result per task, for example:
- `authority_respected`: true or false - `tier_correct`: true or false - `handover_complete`: true or false - `conditions_preserved`: true or false - `rule_version_cited`: the version and whether it was current
**Scoring.** Each check can carry a weight, and the pack can report both a task-level pass and the individual checks so failures are diagnosable.
When licensing onward, keep the tool names and result keys stable across releases, and tell buyers which hooks they can rely on in the version they hold.
Standard hooks and structured grader output let your customers plug a task pack into their evaluation harness without bespoke glue code.
- Audience:
- Licensed resellers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- Which failure modes should an escalation evaluation cover?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?