Skip to content
MirrorFirm

How can an environment evaluate retries and duplicate policy notifications?

Controlled notification delivery and event hooks can expose duplicate side effects, lost updates and premature completion during policy propagation.

A simulated environment can assign each policy notification a stable `change_event_id` and each attempted action a separate `attempt_id`. Delivery controls can repeat a notification, delay an acknowledgement or interrupt a tool response after the underlying update succeeds.

Require mutating actions to accept an `idempotency_key` whose scope is documented. The evaluator can then distinguish a legitimate retry from a second business action. Duplicate evidence requests or repeated review assignments should appear as separate side effects in the trace rather than disappearing into a final-state summary.

Useful hooks include `notification_delivered`, `action_committed`, `response_interrupted` and `run_finished`. Assertions can inspect both intermediate events and terminal state, including whether the agent declared completion while a required acknowledgement remained outstanding.

Reset should restore business records, queued notifications, acknowledgement state and deduplication storage from the same snapshot. Otherwise, one run can influence the next through a hidden delivery history. A declared delivery seed can make failure sequences repeatable, allowing evaluators to compare agent recovery under the same interruption without confusing environmental variation with changes in reasoning.

Test whether repeated or interrupted delivery produces one correct business outcome and a trace evaluators can explain.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Policy change propagation