Skip to content
MirrorFirm

How do I wire scoring and reset into a condition-gated task?

Scoring works best with pre-action and post-action hooks, a per-action JSONL run record, snapshot-based reset and decision-level splits between training and held-out sets.

Scoring a condition-gated task works best when the harness records state changes as they happen, not only the final answer.

**Hooks.** Add a pre-action hook on release_step and any other action that moves a case forward. It logs the status of every blocking condition at that moment. Add a post-action hook on status writes that stores the actor, the old status, the new status and the evidence_id cited.

**Run record.** Write one JSONL line per action: run_id, task_id, step_index, action, arguments, open_blocking_conditions and timestamp. You can then compute premature release by filtering for release_step lines where open_blocking_conditions is not empty.

**Grader inputs.** Compare the final state to the task's expected_end_state: each condition status, the downstream step state and any escalation raised. Compare reason_code values against an accepted set, because some cases allow more than one defensible reason.

**Reset.** Each task points to a start_state_ref, a snapshot of records and inbox contents. Reset restores the snapshot and a fixed seed for any arrival timing, so reruns and model comparisons start from identical conditions.

**Splits.** Keep a held-out set of decisions, not just held-out rows, so related conditions and evidence do not leak between training and test.

**Metrics.** Report premature release, over-blocking, wrong-link evidence, unsupported waiver and correct escalation as separate columns, so one score does not hide the failure type.

Logging open conditions at the moment of each action turns premature release into a simple filter over the run record.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Approval conditions