Which evaluation hooks reveal how an agent assembled its evidence?
Action traces and evaluator-only reference state can distinguish a correct final package from one produced through prohibited actions or unsupported assumptions.
Instrument tool calls with `run_id`, `action_id`, `tool_name`, `arguments`, `result_status` and `state_revision`. Keep these traces separate from the final package so evaluators can inspect how the agent reached its answer. Record denied actions as well as successful ones.
Evaluator-only state can define required findings, prohibited mutations and facts intentionally withheld from the agent. It must not appear in search results, tool descriptions or error messages. Otherwise the environment tests access to the answer rather than evidence reasoning.
Use hooks at meaningful boundaries: after retrieval, after a proposed package update and at final submission. A hook can detect whether the agent modified a source record, submitted an unsupported finding or used information unavailable at its current state revision.
Reset should clear agent-created findings, cached tool results and pending actions while restoring the agreed seed state. Record the reset identifier with the trace. This makes a failed run reproducible and distinguishes an agent error from contamination left by a previous attempt.
Measure process behaviour alongside final output while preventing evaluator information from leaking into the agent's tools.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can an evidence task test resistance to instructions inside retrieved records?
- Which quality checks reveal shortcuts that could undermine an evidence dataset?
- Must an agent's assessment match one reference answer exactly?
- How do you evaluate whether an agent selected the right cases for review?
- What quality checks should precede an agent evaluation?
- How should an evaluation handle an agent that discovers its own earlier mistake?