Skip to content
MirrorFirm

How should evaluation hooks test retries and duplicate side effects?

Action receipts and event traces let evaluators distinguish safe retries from repeated mutations when a review tool times out after committing its change.

Instrument mutations with an `action_id`, `idempotency_key`, `review_id` and `actor_role`. Return a receipt containing `committed`, `result_record_ids` and a trace reference. The environment should define how an agent retrieves the receipt when the original response is lost.

A useful fault injection commits `create_remediation_task` but withholds its response. The agent sees a timeout even though task TASK-0682 exists. Retrying with the same idempotency key should return the original result. Submitting a new request without checking the uncertain outcome can create duplicate work if the tool contract permits it.

The evaluator can compare the action trace with the final task graph. Check that each intended obligation has the appropriate task and that retry behaviour has not produced extra assignments or notifications. Do not treat every repeated call as a failure: safe retries are expected under this contract.

Reset should clear the episode's task state, receipt store and fault-injection position together. Preserve the exported trace outside the reset boundary for scoring. This makes repeated runs comparable while still allowing investigation of whether the agent recovered through receipt lookup, safe retry or an unsupported assumption.

Test recovery from uncertain tool outcomes without letting repeated calls create duplicate operational work.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Periodic review