How can an environment evaluate recovery when referral submission times out?
A timeout after a successful write tests whether an agent checks persistent state and uses idempotency controls instead of creating duplicate specialist requests.
A simulated environment can make `submit_referral` persist a request while returning a timeout to the agent. This creates an ambiguous outcome: the agent does not know whether the action failed before or after the write.
Expose an `idempotency_key` on submission and a lookup action such as `find_referral_by_request`. The generated tool contract should explain their behaviour. A competent recovery path checks for the existing request or retries with the same key, according to that contract. Blindly submitting a fresh request can create a duplicate.
Evaluation hooks can record `request_id`, `idempotency_key`, `persisted_referral_id`, returned error and subsequent actions. The scorer should inspect durable state, not merely count tool errors or retries. A timeout can occur in a successful run if the agent recovers correctly.
Reset must clear the referral store, deduplication registry and injected fault state together. Otherwise, one run can affect another. A fixed fault schedule makes comparisons reproducible, while separately configured variants can test recovery across different failure points without changing the customer's underlying need.
Evaluate reliable recovery from ambiguous tool outcomes, including whether the agent avoids duplicate referrals.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How should evaluation handle a customer changing information during referral submission?
- How can buyers test whether an agent understands referral triggers rather than keywords?
- How can a buyer test resistance to instructions hidden inside customer records?
- What should buyers inspect before using generated hardship material for training?
- Which evaluation hooks reveal duplicate referrals after tool timeouts?
- How do we check that the environment is not giving the agent the answer?