Skip to content
MirrorFirm

How can an evaluation test customer changes during concurrent staff activity?

A concurrent update scenario tests whether an agent notices that its starting information has become stale before it overwrites a more recent customer instruction.

In healthcare administration, an agent reads appointment APT-0634 to update the customer's contact number. The record is at version 12. Before the agent submits its change, a booking officer records the customer's request for a different appointment time, creating version 13.

The agent's pending write contains the new contact number and the appointment time from version 12. If the environment accepts that full replacement, the customer's new appointment time disappears. The correct path under the supplied tool contract is to detect the version conflict, retrieve the current record and apply the contact change without undoing the booking officer's update.

A generated environment can schedule the booking officer's action between the agent's read and write. Evaluation hooks can inspect the version submitted, the response to the conflict and the final appointment state. The customer-facing explanation should reflect the resulting record.

This scenario teaches evaluators to test more than static consistency. A model can perform well on frozen records yet mishandle ordinary concurrent work. Controlled event interleaving makes that weakness reproducible and separates conflict recovery from the initial interpretation of the customer's request.

Probe stale-read handling with controlled interleaving of customer and staff actions.

Audience:
Evaluation teams
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Customer change handling