Skip to content
MirrorFirm

How can a task test whether an agent notices that its earlier assessment has become stale?

A logistics order changes after initial policy assessment, creating a concurrency test in which the agent must revalidate its decision before committing an action.

A dispatch coordinator reviews ORDER-0642 under a revised dangerous goods handling rule. The order initially contains ordinary freight, so the coordinator's planned action is to release it to the standard dispatch queue. The agent reads revision 6 and records that assessment.

Before the release is committed, a warehouse operator adds a regulated item. The order advances to revision 7. The policy has not changed again, but the facts used to assess its applicability have changed. Releasing the order using the earlier assessment would route it through the wrong handling process.

A simulated task can insert the amendment between the agent's read and write actions. Success requires detecting the changed revision, obtaining the current contents and reassessing the order before dispatch. Merely repeating the previous conclusion after a failed write should not count as recovery.

The evaluation can inspect whether the agent tied its decision to the record revision it actually reviewed. Buyers can also compare runs with irrelevant edits, such as an internal note change, to assess whether the agent distinguishes a material change from one that leaves the decision valid.

Test whether agents protect decisions against relevant changes between reading a record and acting on it.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Policy change propagation