Which runtime hooks make policy propagation failures diagnosable?
Runtime hooks can record policy context at every action and evaluate invariants throughout an episode, exposing unsafe intermediate states hidden by final-state scoring.
Instrument each tool call with an action trace containing `episode_id`, `action_id`, `actor_role`, `target_id`, `policy_version_used`, `pre_state_hash`, `post_state_hash` and `result_code`. Keep evaluator-only context separate from tool responses so instrumentation does not reveal hidden answers.
Run invariant checks after state-changing actions. A utility environment might require every confirmed connection to reference a completed assessment whenever the governing policy demands one. If an agent confirms the connection and later repairs the assessment link, a final-state check alone could miss the period of invalid operation.
Useful hooks include `on_action_completed`, `on_policy_activated` and `on_episode_finished`. They can emit structured violations with the affected entity, governing clause and first failing action. An episode-level report can distinguish a prevented invalid attempt from a violation that actually entered state.
MirrorFirm could generate invariant definitions and expected trace conditions alongside the environment specification. Builders should establish which checks their runtime enforces and which only score behaviour. That separation matters: a rejected action is evidence about agent judgement, while an accepted invalid transition may also expose a simulator enforcement gap.
Detect temporary policy violations and explain their causes through structured action evidence.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a buyer inspect business grounding and disclosure risk in generated material?
- What evaluation hooks help test recovery from partial policy-update failures?
- How should evaluation distinguish a correct final state from a correct agent process?
- How can a policy-change scenario test whether an agent respects concurrent edits?
- Which checks reveal whether an agent understands applicability rather than memorising cues?
- How do we evaluate whether an agent changed the right work?