How can an evaluation test safe retries after an uncertain tool response?
A simulated timeout after a successful transfer lets developers test whether retries duplicate downstream work or correctly recover the existing result.
A transfer tool can complete its write and then lose the response. The agent sees a timeout, while the receiving team already has a new task. Retrying without checking can create duplicate work even though both tool calls appear reasonable in isolation.
A simulated environment can expose `create_handover` with `source_record_id`, `destination_queue` and `idempotency_key`. A companion `get_action_status` tool can return the committed result for that key. The generated episode can inject a response failure after the first write, leaving the agent to recover from uncertainty.
Evaluation hooks should inspect the number of downstream tasks associated with the business action, not merely the number of tool attempts. They can also record whether the agent reused the original key and retrieved the existing result before attempting a different action.
Reset should clear the episode's tasks and action ledger together, while preserving the chosen fault-injection configuration. That makes repeated runs comparable. The test asks a concrete question: can the agent recover an uncertain outcome without multiplying obligations for the receiving team?
Measure duplicate side effects under uncertain responses, using stable action identifiers and observable downstream state.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- When should escalation count as a successful agent outcome?
- What should buyers inspect before trusting a generated handover sample?
- How can an evaluation expose decisions based on stale evidence?
- How should handover evaluation go beyond checking task completion?
- How can we tell whether an agent learned the handover rule rather than a routing shortcut?
- What should we inspect before using generated handover material in training?