Skip to content
MirrorFirm

What grader outputs and evaluation hooks could a task pack provide for gated release?

Technical detail on how a task pack can express expected outcomes, scoring fields and failure labels so evaluation results are comparable across agents and runs.

A task pack could include a task file, expected-state definitions and a grader that writes a result per run.

**Task file (JSONL).** Fields such as `task_id`, `snapshot_id`, `instruction`, `allowed_actions`, `difficulty` and `expected_outcome` (for example `hold_and_escalate`, `release`, `request_evidence`).

**Expected state.** A small definition of what the final tables should show: which holds exist, which releases must not exist, which messages should be sent and to which role.

**Grader output (JSON per run).** `run_id`, `task_id`, `passed`, `score`, `failure_labels` and `evidence_event_ids`. Failure labels might include `release_with_open_blocking_condition`, `wrong_role_notified`, `condition_marked_satisfied_without_review` and `ignored_variation`.

**Hooks.** A pre-run hook loads the snapshot. A post-run hook reads final tables and the event log. An optional trace hook records the agent's tool calls for later review.

**Aggregation.** Results can be rolled up by label and difficulty, so a buyer sees which kinds of failure dominate.

Labels, fields and scoring rules are agreed per pack, so ask for a worked example with one passing and one failing run. The tasks and states are generated and synthetic.

Structured grader outputs let a reseller's customers compare agents and prompts consistently, without writing their own scoring logic first.

Audience:
Licensed resellers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Approval conditions