Which evaluation hooks can verify what the agent actually changed?
Hidden state assertions and action traces can verify operational outcomes independently of an agent's explanation, including duplicate actions, prohibited transitions and unsupported completion claims.
An evaluation environment can expose actions such as `read_policy`, `list_work`, `request_evidence` and `apply_transition`, while keeping expected outcomes in an evaluator-only store. The agent's final answer should be assessed alongside the state produced by those actions.
Each action trace can record `run_id`, `action_id`, `target_id`, `input_revision`, `result_revision` and `outcome_code`. Hidden assertions can check that required evidence exists before a transition, that protected records retain their state and that completion claims correspond to committed changes. Assertions should refer to business conditions rather than demand one exact sequence where several sequences are valid.
Side effects need inspection too. A retry that sends duplicate evidence requests may leave the main record correct while still producing an operational failure. Idempotency keys and a generated outbound queue can make this visible to the evaluator.
Reset hooks should restore both primary records and side-effect stores, including pending requests and scheduled events. Otherwise, one run can influence the next. A buyer can ask for a reset verification result before each evaluation and retain the final state diff with the action trace for failure analysis.
Score actual state transitions and side effects rather than relying on a persuasive final response.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a buyer inspect business grounding and disclosure risk in generated material?
- What evaluation hooks help test recovery from partial policy-update failures?
- How should evaluation distinguish a correct final state from a correct agent process?
- How can a policy-change scenario test whether an agent respects concurrent edits?
- Which checks reveal whether an agent understands applicability rather than memorising cues?
- How do we evaluate whether an agent changed the right work?