Skip to content
MirrorFirm

Which evaluation hooks make evidence assembly results inspectable?

Action traces and replayable transitions let evaluators inspect how an agent reached its package outcome, including invalid actions hidden by a plausible final response.

Record each tool attempt with `episode_id`, `step_id`, `action_name`, `argument_ref`, `result_code` and references to the relevant before and after state. Capture rejected attempts as well as successful mutations, because an unauthorised action can matter even when the environment blocks it.

Provide hooks such as `on_evidence_linked`, `on_exception_changed` and `on_package_submitted`. Each hook can validate a specific condition: whether the cited record exists, whether the acting role may close the exception, or whether required assertions have an explicit assessment.

Keep scoring dimensions separate. A package can have accurate conclusions but unresolved citations, or complete citations but an unsupported claim. Reporting those outcomes independently gives builders more useful feedback than a single opaque score.

Replay should begin from an identified reset fixture and apply the recorded actions under the same transition rules. Evaluator-only labels should remain outside agent-visible responses and exports used during the episode. Buyers can inspect a replay to establish whether success came from permitted interactions. This also exposes cases where a polished final answer conceals a failed submission, a blocked mutation or evidence that was never actually attached.

Score observable state changes and evidence support alongside the submitted explanation.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Audit evidence assembly