Where do reset, trace and grading hooks fit around an agent run?
Sets out evaluation wiring before, during and after a run: state reset and hashing, JSONL tool traces with change events, deterministic checks and calibrated note grading.
Wire evaluation in at three points: before the run, during it, and after it.
**Before.** Call `reset(case_id)` to restore the initial state, then record a hash of the state so you can confirm repeated runs start identically. Load only `cases`, `records` and `policies` into the agent's reach; keep `labels` on the grader side.
**During.** Log every tool call as a JSONL trace line: `run_id`, `case_id`, `step`, `tool`, `arguments`, `result_status`, `timestamp`. Write actions should emit a change event with the record ID and before and after values, so the grader can reconstruct what was altered without re-reading the whole state.
**After.** Run deterministic checks: - final state diff against the expected state or the set of acceptable end states; - any call on the forbidden-action list; - whether the evidence records were read before the first write; - routing target for approvals.
Then score the note against `expected_note_elements`, by rubric or by a model grader that you calibrate against human review.
Report per-component scores (cause, action, evidence, routing, note) and per-failure-mode tallies rather than one number. That makes it clear whether a change to your agent improved investigation or only made it better at forcing totals to agree.
Ask for the trace schema in the pilot so your harness and the pack agree on field names.
Per-component scores and failure-mode breakdowns show whether a change improved investigation or only improved the agent at forcing totals to agree.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How does path-level scoring separate two agents that both match the totals?
- How can a buyer inspect the quality of generated mismatch cases?
- What checks should I run on a pack before using it for evaluation?
- How should I score an agent that reaches the right fix for the wrong reason?
- How do we know a generated mismatch case is solvable from what the agent can see?
- What evaluation hooks should a mismatch environment expose for reset, traces and scoring?