Which hooks reveal duplicate effects when a change action is retried?
A runner can inject uncertain tool responses and inspect idempotency records to determine whether retries duplicate customer notifications, adjustments or downstream change requests.
A simulated tool can commit a change and then return a timeout. The agent must decide whether to retry, inspect status or escalate. This tests a different problem from an ordinary rejected request: the business action may already have occurred.
Define mutating actions with an `operation_id` or `idempotency_key`, plus a status lookup such as `get_operation_status`. An action log can record `attempt_id`, `operation_id`, `committed_at` and `effect_ids`. Separate attempts from effects so the evaluator can detect a duplicate notification or adjustment even when the final customer field looks correct.
The runner can inject the timeout after commit at a controlled point. Scoring hooks can check whether the agent reused the operation key, consulted available status information and avoided creating an additional business effect. The expected recovery path should follow the supplied tool semantics.
Reset must clear operation registries, outbound queues and action logs as well as customer records. Otherwise, an earlier run's idempotency state can affect a later result. Request a reset assertion that checks those stores before each evaluation episode begins.
Test recovery from uncertain execution without confusing a repeated call with a repeated business effect.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?