Which evaluation hooks expose retry and duplicate-processing failures?
Deterministic resets, injected tool failures and observable side effects let developers test whether retries complete a change without producing duplicate business actions.
A simulated environment can expose `reset(initial_state_id, seed)` and retain an append-only action log for each run. Reset semantics should restore records, pending messages, simulated time and deduplication state; restoring only database rows can leave later runs contaminated by earlier activity.
A failure hook can make `submit_change` commit successfully while its response times out. The agent then faces an uncertain outcome. A sound recovery uses an operation identifier or a status lookup before deciding whether another submission is needed.
Evaluation hooks can count business side effects by `operation_id`, inspect `idempotency_key` reuse and distinguish repeated tool calls from duplicated actions. Calling a tool twice may be harmless; creating two replacement delivery bookings is not.
Buyers can request failure schedules that are reproducible and hidden from the agent. The harness can record the injection point alongside the action trace for diagnosis. This makes it possible to assess whether recovery preserves exactly the intended business outcome even when transport behaviour gives the agent incomplete information about what happened.
Measure recovery at the side-effect boundary, where an uncertain tool response can conceal a successful write.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?
- What should buyers inspect before trusting a generated evaluation pack?