How can an evaluation test customer changes during concurrent staff activity?
A concurrent update scenario tests whether an agent notices that its starting information has become stale before it overwrites a more recent customer instruction.
In healthcare administration, an agent reads appointment APT-0634 to update the customer's contact number. The record is at version 12. Before the agent submits its change, a booking officer records the customer's request for a different appointment time, creating version 13.
The agent's pending write contains the new contact number and the appointment time from version 12. If the environment accepts that full replacement, the customer's new appointment time disappears. The correct path under the supplied tool contract is to detect the version conflict, retrieve the current record and apply the contact change without undoing the booking officer's update.
A generated environment can schedule the booking officer's action between the agent's read and write. Evaluation hooks can inspect the version submitted, the response to the conflict and the final appointment state. The customer-facing explanation should reflect the resulting record.
This scenario teaches evaluators to test more than static consistency. A model can perform well on frozen records yet mishandle ordinary concurrent work. Controlled event interleaving makes that weakness reproducible and separates conflict recovery from the initial interpretation of the customer's request.
Probe stale-read handling with controlled interleaving of customer and staff actions.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?