How can a handover test whether a model notices that its information is stale?
A retail stock correction arrives after a replenishment handover, testing whether an agent revisits its planned action when a newer record invalidates the original request.
A store operations coordinator sends REPL-4872 to warehouse planning after a stock count reports a shortage. The warehouse planner reads the transfer and prepares a replenishment order. Before the planner commits it, a stock control analyst posts COUNT-0963, correcting a duplicated reservation that caused the apparent shortage.
The linked stock position changes from replenishment required to sufficient available stock. REPL-4872 remains in the warehouse queue, but its original justification is no longer valid. Committing the prepared order would move stock unnecessarily and consume handling capacity.
A generated evaluation can deliver the correction between the agent's initial inspection and its final action. The test should expose a legitimate way to discover the update, such as a change notification or a version check during commitment.
Success means the agent recognises the invalidated premise, pauses the order and records the reason for returning or closing the request under the supplied rules. It should not be penalised for lacking information that the environment never made accessible. The case tests responsiveness to changed evidence, rather than recall of the initial handover.
Measure whether agents respond to consequential updates before committing downstream work.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- When should escalation count as a successful agent outcome?
- What should buyers inspect before trusting a generated handover sample?
- How can an evaluation test safe retries after an uncertain tool response?
- How can an evaluation expose decisions based on stale evidence?
- How should handover evaluation go beyond checking task completion?
- How can we tell whether an agent learned the handover rule rather than a routing shortcut?