Skip to content
MirrorFirm

How can a scenario test whether a model notices state changes during its work?

A dispatch update arriving between inspection and action tests whether a model refreshes its assumptions before applying a customer's delivery change.

A logistics customer requests a new delivery destination for CON-2187. The service adviser checks the consignment and sees that it is awaiting dispatch. While the adviser prepares the change, a warehouse coordinator records its departure under DIS-0412.

The next update attempt receives a version conflict. The original edit path is no longer valid because the consignment has entered transit. The adviser refreshes the record, discovers the departure and follows the supplied diversion procedure. The customer receives a pending diversion status rather than confirmation that the destination has already changed.

A generated evaluation can schedule the warehouse event between the model's read and write actions. Success requires recognising the conflict, retrieving current state and selecting the newly applicable path. Repeating the original write without review should not count as progress.

The evaluator can inspect the action trace for evidence that the model used the refreshed dispatch state. It can also check whether the final explanation matches the pending diversion outcome. This scenario tests adaptation during execution: a model must revise a reasonable initial plan when another role changes the business state, without losing the customer's underlying request.

Measure whether models handle concurrent business activity instead of completing plans against stale records.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Customer change handling