How should evaluation handle a customer changing information during referral submission?
A customer update arriving during submission tests whether an agent detects stale case state and preserves the latest instructions in the specialist handover.
A retail payment support agent prepares referral REF-4815 using case CASE-0936, version 4. The customer has requested a telephone callback. Before submission completes, a service adviser records a new message: the customer can now communicate only through the account portal. CASE-0936 advances to version 5.
The agent submits its earlier draft with expected_case_version set to 4. The environment rejects the write as stale and returns the current version identifier. The agent retrieves the update, revises the contact instructions and resubmits against version 5.
The resulting referral carries the portal preference. If the agent ignores the conflict or blindly repeats the earlier payload, the specialist team receives an outdated contact route and may be unable to reach the customer appropriately.
This evaluation examines recovery from a meaningful state change. The scoring hook can inspect whether the agent fetched the new version, preserved the customer's latest instruction and avoided overwriting the intervening note. A fluent apology is insufficient if the persisted referral still contains stale information. The task teaches developers to assess write correctness under concurrent activity.
Test concurrency handling where an otherwise correct referral becomes outdated before it is accepted.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can buyers test whether an agent understands referral triggers rather than keywords?
- How can a buyer test resistance to instructions hidden inside customer records?
- What should buyers inspect before using generated hardship material for training?
- Which evaluation hooks reveal duplicate referrals after tool timeouts?
- How do we check that the environment is not giving the agent the answer?
- How can evaluation hooks catch consequences that occur after the conversation ends?