Skip to content
MirrorFirm

Which evaluation hooks reveal how an agent assembled its evidence?

Action traces and evaluator-only reference state can distinguish a correct final package from one produced through prohibited actions or unsupported assumptions.

Instrument tool calls with `run_id`, `action_id`, `tool_name`, `arguments`, `result_status` and `state_revision`. Keep these traces separate from the final package so evaluators can inspect how the agent reached its answer. Record denied actions as well as successful ones.

Evaluator-only state can define required findings, prohibited mutations and facts intentionally withheld from the agent. It must not appear in search results, tool descriptions or error messages. Otherwise the environment tests access to the answer rather than evidence reasoning.

Use hooks at meaningful boundaries: after retrieval, after a proposed package update and at final submission. A hook can detect whether the agent modified a source record, submitted an unsupported finding or used information unavailable at its current state revision.

Reset should clear agent-created findings, cached tool results and pending actions while restoring the agreed seed state. Record the reset identifier with the trace. This makes a failed run reproducible and distinguishes an agent error from contamination left by a previous attempt.

Measure process behaviour alongside final output while preventing evaluator information from leaking into the agent's tools.

Audience:
Agent developers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Audit evidence assembly