Skip to content
MirrorFirm

What evaluation hooks and grader outputs can a task pack provide?

A task pack could expose tool actions, a reset command, a final state diff and per-check results, so buyers can run repeatable evaluations in their own harness.

A task pack for this workflow can define a small set of hooks.

**Tool actions.** Typical actions include `get_case`, `lookup_authority`, `get_policy`, `escalate_case`, `record_decision`, `add_condition` and `request_document`. Each has a typed input and a result, so an agent's calls can be logged and replayed.

**Reset.** A `reset(task_id, seed)` call restores the environment to the snapshot named by `initial_state_ref`. With the same seed, a run starts from identical state, which makes comparisons between model versions meaningful.

**Final state diff.** After a run, the grader can output the difference between actual and expected state, limited to the tables the task touches.

**Check results.** A structured result per task, for example:

- `authority_respected`: true or false - `tier_correct`: true or false - `handover_complete`: true or false - `conditions_preserved`: true or false - `rule_version_cited`: the version and whether it was current

**Scoring.** Each check can carry a weight, and the pack can report both a task-level pass and the individual checks so failures are diagnosable.

When licensing onward, keep the tool names and result keys stable across releases, and tell buyers which hooks they can rely on in the version they hold.

Standard hooks and structured grader output let your customers plug a task pack into their evaluation harness without bespoke glue code.

Audience:
Licensed resellers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Credit escalation