Skip to content
MirrorFirm

Which evaluation hooks make review-agent behaviour inspectable?

Evaluation hooks can score required actions, prohibited actions and evidence-supported outcomes from event traces, while allowing different valid action sequences to satisfy the same review task.

Capture an event trace with `run_id`, `event_id`, `action`, `target_id`, `result` and `state_revision`. Link decisions to their cited `evidence_ids`. The evaluator can then inspect what the agent attempted, what the environment accepted and what state remained at termination.

Represent task expectations as obligations rather than one exact transcript. An obligation might require an evidence request before a decision, while a prohibition prevents closure with an unresolved blocking finding. Several valid action sequences can satisfy the same requirements.

Useful hooks include `on_action`, `on_state_change` and `on_run_end`. The final hook can check unresolved obligations, orphaned follow-up cases and whether the submitted explanation cites accessible evidence. Record component results separately so customers can distinguish a reasoning failure from a tool execution failure.

Keep evaluator-only rules outside the agent's retrieval scope. Version the scoring contract alongside the task pack, and document how retries or rejected actions affect scoring. For resellers, that contract enables customers to reproduce the evaluation without depending on a proprietary transcript format or assuming that every deviation from a reference sequence is an error.

Resellers can provide transparent scoring contracts that customers can audit and adapt to their own agent implementations.

Audience:
Licensed resellers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Periodic review