How should evaluation hooks test retries and duplicate side effects?
Action receipts and event traces let evaluators distinguish safe retries from repeated mutations when a review tool times out after committing its change.
Instrument mutations with an `action_id`, `idempotency_key`, `review_id` and `actor_role`. Return a receipt containing `committed`, `result_record_ids` and a trace reference. The environment should define how an agent retrieves the receipt when the original response is lost.
A useful fault injection commits `create_remediation_task` but withholds its response. The agent sees a timeout even though task TASK-0682 exists. Retrying with the same idempotency key should return the original result. Submitting a new request without checking the uncertain outcome can create duplicate work if the tool contract permits it.
The evaluator can compare the action trace with the final task graph. Check that each intended obligation has the appropriate task and that retry behaviour has not produced extra assignments or notifications. Do not treat every repeated call as a failure: safe retries are expected under this contract.
Reset should clear the episode's task state, receipt store and fault-injection position together. Preserve the exported trace outside the reset boundary for scoring. This makes repeated runs comparable while still allowing investigation of whether the agent recovered through receipt lookup, safe retry or an unsupported assumption.
Test recovery from uncertain tool outcomes without letting repeated calls create duplicate operational work.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should precede training on generated review cases?
- How can a late event reveal whether an agent checks its assumptions before committing a decision?
- Which evaluation hooks make review runs reproducible and auditable?
- How should evaluation distinguish a sound review decision from a persuasive explanation?
- How can we tell whether an agent responds to meaningful evidence rather than superficial cues?
- How should buyers inspect realism and disclosure risk in generated review material?