How can tool hooks test recovery from a transfer request with an uncertain outcome?
Controlled response loss can test whether an agent checks transfer state before retrying an action that may already have succeeded.
A simulated transfer tool could commit an action and then withhold its success response. The agent sees a timeout, while the receiving queue already contains the new work item. This creates an uncertain outcome rather than an ordinary rejected request.
Give `create_transfer` an `idempotency_key` and provide a separate `get_transfer_status` action. Log `request_id`, `idempotency_key`, `commit_event_id` and `response_delivered` in evaluator-only telemetry. The agent can then resolve uncertainty through a status check or a retry that preserves the original key.
An evaluation hook should detect duplicate receiving tasks, conflicting transfer contents and unsupported claims that the action failed. It should also distinguish a safe repeated request from a second business operation created with a fresh key.
Reset the environment to the same initial snapshot and fault schedule for each compared run. Keep the hidden commit status out of the agent's prompt. This makes recovery behaviour reproducible while requiring the agent to use available tools to establish what happened before it creates more work.
Evaluate recovery behaviour without rewarding duplicate work creation.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- When should escalation count as a successful agent outcome?
- What should buyers inspect before trusting a generated handover sample?
- How can an evaluation test safe retries after an uncertain tool response?
- How can an evaluation expose decisions based on stale evidence?
- How should handover evaluation go beyond checking task completion?
- How can we tell whether an agent learned the handover rule rather than a routing shortcut?