How can a policy-change task test safe handling of concurrent updates?
A concurrent case update tests whether an agent refreshes its reasoning before applying a policy change to state that has moved.
A credit officer opens application `APP-2187` at revision `REV-0012`. The application awaits a decision, and a new policy requires additional income evidence before approval. The agent prepares an update that adds the evidence requirement and places the application on hold.
Before the update is submitted, an authorised reviewer withdraws the application at revision `REV-0013` following a recorded withdrawal request. The agent's proposed hold now targets stale state. Applying it without checking the revision would reopen work that should remain closed.
A generated environment can reject the stale write and return a revision conflict. The agent must reload the application, recognise that the withdrawal changes the appropriate action and abandon the obsolete update. It can retain an audit explanation without changing the withdrawn status.
The evaluation checks recovery after the conflict, not merely whether the tool rejected the write. Repeatedly resubmitting the same action or forcing a replacement remains a failure. The scenario teaches that correct policy reasoning must be refreshed when another authorised action changes the business object to which that reasoning applies.
Reveal agents that overwrite newer business decisions while correctly interpreting the policy itself.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a buyer inspect business grounding and disclosure risk in generated material?
- What evaluation hooks help test recovery from partial policy-update failures?
- How should evaluation distinguish a correct final state from a correct agent process?
- How can a policy-change scenario test whether an agent respects concurrent edits?
- Which checks reveal whether an agent understands applicability rather than memorising cues?
- How do we evaluate whether an agent changed the right work?