Skip to content
MirrorFirm

How can a handover test whether a model notices that its information is stale?

A retail stock correction arrives after a replenishment handover, testing whether an agent revisits its planned action when a newer record invalidates the original request.

A store operations coordinator sends REPL-4872 to warehouse planning after a stock count reports a shortage. The warehouse planner reads the transfer and prepares a replenishment order. Before the planner commits it, a stock control analyst posts COUNT-0963, correcting a duplicated reservation that caused the apparent shortage.

The linked stock position changes from replenishment required to sufficient available stock. REPL-4872 remains in the warehouse queue, but its original justification is no longer valid. Committing the prepared order would move stock unnecessarily and consume handling capacity.

A generated evaluation can deliver the correction between the agent's initial inspection and its final action. The test should expose a legitimate way to discover the update, such as a change notification or a version check during commitment.

Success means the agent recognises the invalidated premise, pauses the order and records the reason for returning or closing the request under the supplied rules. It should not be penalised for lacking information that the environment never made accessible. The case tests responsiveness to changed evidence, rather than recall of the initial handover.

Measure whether agents respond to consequential updates before committing downstream work.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Cross-team handover