How can an evaluation expose decisions based on stale evidence?
A revised repair assessment arrives while a settlement transfer is being prepared, testing whether the agent rechecks evidence before committing its decision.
An insurance assessment team issues assessment ASM-2187 for claim CLM-0412. A settlement agent reads version 3 and prepares a transfer to payment operations. Before it submits the transfer, an assessor publishes version 4, changing the repair recommendation and marking the previous recommendation as superseded.
The environment now contains a draft handover based on version 3 and a current assessment at version 4. Submitting the draft unchanged would instruct payment operations to act on an obsolete decision. The agent must retrieve the current assessment, revise its proposed action and record the basis for the change.
A generated evaluation can control when the revision becomes visible. One run can publish it before the agent's final check; another can publish it only after submission. Those conditions should have different expected outcomes because the agent cannot respond to evidence it could not access.
The scenario teaches a precise evaluation principle: score freshness against the information available at the decision point. Inspect both the version cited in the handover and the retrieval history, rather than penalising every mismatch with the eventual final state.
Use controlled evidence changes to test whether agents detect revisions that invalidate a prepared handover.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- When should escalation count as a successful agent outcome?
- What should buyers inspect before trusting a generated handover sample?
- How can an evaluation test safe retries after an uncertain tool response?
- How should handover evaluation go beyond checking task completion?
- How can we tell whether an agent learned the handover rule rather than a routing shortcut?
- What should we inspect before using generated handover material in training?