How should evaluation handle a destination that changes while the agent is working?
A logistics queue stops accepting requests between lookup and assignment, testing whether the agent handles stale state without losing or duplicating the request.
A logistics intake agent receives REQ-7306 about a missed collection. Its lookup shows queue QUE-2104 accepting work for the relevant depot. The agent prepares an assignment using queue revision REV-0812.
Before the assignment call arrives, a depot coordinator closes that queue to new requests and activates a contingency destination. The environment advances the queue revision. The assignment tool rejects the stale operation and leaves REQ-7306 unassigned.
The agent refreshes the queue state, reads the contingency instruction and submits a valid assignment to the replacement destination. The request then changes to assigned. The agent reports the accepted destination only after checking the result.
A weak response would repeatedly submit the stale action, claim the first call succeeded or create a second request to escape the error. Each behaviour introduces a different operational problem.
A generated evaluation could inject this state change at the tool boundary. Success criteria should cover recovery, preservation of the original request and absence of duplicate assignments. The task teaches that routing correctness depends on current state at execution, not just evidence collected earlier in the conversation.
Evaluate recovery from concurrent state changes, including truthful reporting of whether an assignment actually succeeded.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can buyers test whether an agent uses the right evidence to choose a destination?
- Which evaluation hooks reveal whether a routing action preserved the required handover?
- What should buyers inspect before using a generated routing corpus?
- How can buyers detect answer leakage in generated routing evaluations?
- What should buyers check in generated content before trusting agent scores?
- How can an evaluation expose routing behaviour that keeps work moving without resolving ownership?