How can evaluation detect duplicate referrals caused by tool retries?

A generated environment can simulate a create_referral request that commits successfully but returns a timeout to the model. The persisted referral exists; the visible tool result does not confirm it. This separates business state from transport feedback.

The tool contract can support an idempotency_key and a lookup action keyed by source_contact_id. The supplied procedure should specify how the agent determines whether submission succeeded. Evaluation can then observe whether the model checks existing state, retries using the same key or creates another referral with a new identifier.

Useful hooks include mutation_committed, response_delivered and duplicate_work_item_created. The evaluator should inspect persisted records rather than infer success from the final assistant message. A well-worded response does not compensate for two specialist tasks representing one customer request.

Reset should restore the initial records and clear run-specific idempotency entries. A fixed fault schedule can reproduce the same timeout position across model runs. This gives labs a controlled way to compare recovery behaviour without allowing previous attempts to alter the next evaluation.