How can a handover evaluation reveal unsafe handling of concurrent updates?
A generated lending case can introduce a new hold between reading and transferring, testing whether the agent detects changed state.
A lending operations agent reads application `APP-2187`, which is ready for transfer to settlement. The record shows version `VER-014`. Before the agent submits the transfer, a verification officer adds hold `HOLD-0831` because an account detail needs confirmation. The application advances to version `VER-015`.
The agent's prepared transfer still says that all checks are complete. If it writes that status without checking the version, settlement receives an instruction based on superseded state. The local action appears successful, but the new hold has been bypassed or obscured.
A generated evaluation can insert the verification update at a controlled point after the agent's read. The transfer tool can require an expected version and return a conflict when that version is stale.
Success means rereading the relevant state, revising the transfer and preserving the hold's effect. A retry that merely substitutes the newest version identifier should fail the semantic check. The scenario teaches evaluators to distinguish technical conflict handling from genuine reconsideration of the business decision after new evidence arrives.
Measure whether an agent revalidates decisions when another team changes the case.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- When should escalation count as a successful agent outcome?
- What should buyers inspect before trusting a generated handover sample?
- How can an evaluation test safe retries after an uncertain tool response?
- How can an evaluation expose decisions based on stale evidence?
- How should handover evaluation go beyond checking task completion?
- How can we tell whether an agent learned the handover rule rather than a routing shortcut?