How can an environment evaluate retries and duplicate policy notifications?
Controlled notification delivery and event hooks can expose duplicate side effects, lost updates and premature completion during policy propagation.
A simulated environment can assign each policy notification a stable `change_event_id` and each attempted action a separate `attempt_id`. Delivery controls can repeat a notification, delay an acknowledgement or interrupt a tool response after the underlying update succeeds.
Require mutating actions to accept an `idempotency_key` whose scope is documented. The evaluator can then distinguish a legitimate retry from a second business action. Duplicate evidence requests or repeated review assignments should appear as separate side effects in the trace rather than disappearing into a final-state summary.
Useful hooks include `notification_delivered`, `action_committed`, `response_interrupted` and `run_finished`. Assertions can inspect both intermediate events and terminal state, including whether the agent declared completion while a required acknowledgement remained outstanding.
Reset should restore business records, queued notifications, acknowledgement state and deduplication storage from the same snapshot. Otherwise, one run can influence the next through a hidden delivery history. A declared delivery seed can make failure sequences repeatable, allowing evaluators to compare agent recovery under the same interruption without confusing environmental variation with changes in reasoning.
Test whether repeated or interrupted delivery produces one correct business outcome and a trace evaluators can explain.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a buyer inspect business grounding and disclosure risk in generated material?
- What evaluation hooks help test recovery from partial policy-update failures?
- How should evaluation distinguish a correct final state from a correct agent process?
- How can a policy-change scenario test whether an agent respects concurrent edits?
- Which checks reveal whether an agent understands applicability rather than memorising cues?
- How do we evaluate whether an agent changed the right work?