Skip to content
MirrorFirm

Which evaluation hooks can verify what the agent actually changed?

Hidden state assertions and action traces can verify operational outcomes independently of an agent's explanation, including duplicate actions, prohibited transitions and unsupported completion claims.

An evaluation environment can expose actions such as `read_policy`, `list_work`, `request_evidence` and `apply_transition`, while keeping expected outcomes in an evaluator-only store. The agent's final answer should be assessed alongside the state produced by those actions.

Each action trace can record `run_id`, `action_id`, `target_id`, `input_revision`, `result_revision` and `outcome_code`. Hidden assertions can check that required evidence exists before a transition, that protected records retain their state and that completion claims correspond to committed changes. Assertions should refer to business conditions rather than demand one exact sequence where several sequences are valid.

Side effects need inspection too. A retry that sends duplicate evidence requests may leave the main record correct while still producing an operational failure. Idempotency keys and a generated outbound queue can make this visible to the evaluator.

Reset hooks should restore both primary records and side-effect stores, including pending requests and scheduled events. Otherwise, one run can influence the next. A buyer can ask for a reset verification result before each evaluation and retain the final state diff with the action trace for failure analysis.

Score actual state transitions and side effects rather than relying on a persuasive final response.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Policy change propagation