Which evaluation hooks make evidence assembly results inspectable?
Action traces and replayable transitions let evaluators inspect how an agent reached its package outcome, including invalid actions hidden by a plausible final response.
Record each tool attempt with `episode_id`, `step_id`, `action_name`, `argument_ref`, `result_code` and references to the relevant before and after state. Capture rejected attempts as well as successful mutations, because an unauthorised action can matter even when the environment blocks it.
Provide hooks such as `on_evidence_linked`, `on_exception_changed` and `on_package_submitted`. Each hook can validate a specific condition: whether the cited record exists, whether the acting role may close the exception, or whether required assertions have an explicit assessment.
Keep scoring dimensions separate. A package can have accurate conclusions but unresolved citations, or complete citations but an unsupported claim. Reporting those outcomes independently gives builders more useful feedback than a single opaque score.
Replay should begin from an identified reset fixture and apply the recorded actions under the same transition rules. Evaluator-only labels should remain outside agent-visible responses and exports used during the episode. Buyers can inspect a replay to establish whether success came from permitted interactions. This also exposes cases where a polished final answer conceals a failed submission, a blocked mutation or evidence that was never actually attached.
Score observable state changes and evidence support alongside the submitted explanation.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which evaluation hooks reveal how an agent assembled its evidence?
- How can an evidence task test resistance to instructions inside retrieved records?
- Which quality checks reveal shortcuts that could undermine an evidence dataset?
- Must an agent's assessment match one reference answer exactly?
- How do you evaluate whether an agent selected the right cases for review?
- What quality checks should precede an agent evaluation?