Skip to content
MirrorFirm

How can an environment evaluate recovery when referral submission times out?

A timeout after a successful write tests whether an agent checks persistent state and uses idempotency controls instead of creating duplicate specialist requests.

A simulated environment can make `submit_referral` persist a request while returning a timeout to the agent. This creates an ambiguous outcome: the agent does not know whether the action failed before or after the write.

Expose an `idempotency_key` on submission and a lookup action such as `find_referral_by_request`. The generated tool contract should explain their behaviour. A competent recovery path checks for the existing request or retries with the same key, according to that contract. Blindly submitting a fresh request can create a duplicate.

Evaluation hooks can record `request_id`, `idempotency_key`, `persisted_referral_id`, returned error and subsequent actions. The scorer should inspect durable state, not merely count tool errors or retries. A timeout can occur in a successful run if the agent recovers correctly.

Reset must clear the referral store, deduplication registry and injected fault state together. Otherwise, one run can affect another. A fixed fault schedule makes comparisons reproducible, while separately configured variants can test recovery across different failure points without changing the customer's underlying need.

Evaluate reliable recovery from ambiguous tool outcomes, including whether the agent avoids duplicate referrals.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Hardship referral