What grader outputs and evaluation hooks could a task pack provide for gated release?
Technical detail on how a task pack can express expected outcomes, scoring fields and failure labels so evaluation results are comparable across agents and runs.
A task pack could include a task file, expected-state definitions and a grader that writes a result per run.
**Task file (JSONL).** Fields such as `task_id`, `snapshot_id`, `instruction`, `allowed_actions`, `difficulty` and `expected_outcome` (for example `hold_and_escalate`, `release`, `request_evidence`).
**Expected state.** A small definition of what the final tables should show: which holds exist, which releases must not exist, which messages should be sent and to which role.
**Grader output (JSON per run).** `run_id`, `task_id`, `passed`, `score`, `failure_labels` and `evidence_event_ids`. Failure labels might include `release_with_open_blocking_condition`, `wrong_role_notified`, `condition_marked_satisfied_without_review` and `ignored_variation`.
**Hooks.** A pre-run hook loads the snapshot. A post-run hook reads final tables and the event log. An optional trace hook records the agent's tool calls for later review.
**Aggregation.** Results can be rolled up by label and difficulty, so a buyer sees which kinds of failure dominate.
Labels, fields and scoring rules are agreed per pack, so ask for a worked example with one passing and one failing run. The tasks and states are generated and synthetic.
Structured grader outputs let a reseller's customers compare agents and prompts consistently, without writing their own scoring logic first.
- Audience:
- Licensed resellers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?