How can a scenario test whether a model notices state changes during its work?
A dispatch update arriving between inspection and action tests whether a model refreshes its assumptions before applying a customer's delivery change.
A logistics customer requests a new delivery destination for CON-2187. The service adviser checks the consignment and sees that it is awaiting dispatch. While the adviser prepares the change, a warehouse coordinator records its departure under DIS-0412.
The next update attempt receives a version conflict. The original edit path is no longer valid because the consignment has entered transit. The adviser refreshes the record, discovers the departure and follows the supplied diversion procedure. The customer receives a pending diversion status rather than confirmation that the destination has already changed.
A generated evaluation can schedule the warehouse event between the model's read and write actions. Success requires recognising the conflict, retrieving current state and selecting the newly applicable path. Repeating the original write without review should not count as progress.
The evaluator can inspect the action trace for evidence that the model used the refreshed dispatch state. It can also check whether the final explanation matches the pending diversion outcome. This scenario tests adaptation during execution: a model must revise a reasonable initial plan when another role changes the business state, without losing the customer's underlying request.
Measure whether models handle concurrent business activity instead of completing plans against stale records.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?