How should I structure a task file, run log and scorer for repeatable evaluation?

Evaluation hooks turn a task pack into a repeatable measurement. A workable design has four parts.

**Task file** (JSONL): task_id, instruction, snapshot_id, tools_allowed, expected_state_id, rule_tags, difficulty_band.

**Snapshot**: Parquet tables for bank_line, invoice, customer, credit_note, allocation, case and suspense, loaded into the environment at the start of each run.

**Run log** (JSONL): task_id, run_id, step, tool, arguments, result, state_hash.

**Scorer**: a function that takes the final state and the expected state and returns a record with these fields:

- state_match: true or false for the full ledger. - item_errors: list of item_ids with the wrong final balance. - unsafe_actions: count of postings to the wrong customer. - unneeded_escalations: count of cases raised where the reference allocates. - policy_citations: whether the agent's note names the correct rule, if you require notes.

Run each task more than once and store run_id so you can compute consistency. Because rule_tags are on every task, you can group results by rule, such as tolerance handling or reversal recovery.

Reset is done by reloading the snapshot, not by undoing steps, which avoids drift between runs. Ask for the scorer to be provided as readable code or as a specification so you can port it to your harness. Exact field names are agreed during a pilot.