Must an agent follow one reference sequence to pass a handover task?
Handover evaluations can accept different authorised action sequences when they satisfy the same business constraints and leave equivalent responsibility states.
**Should the reference action sequence be the only passing answer?**
Usually not. A generated handover task may permit several legitimate routes. An agent might request a missing attachment before drafting the transfer, or prepare a draft and request the attachment before dispatch. Both can satisfy the same business requirement.
**What should remain fixed?**
Define the invariants: required evidence is present before dispatch, no unauthorised approval occurs, unresolved work retains an owner and the final state matches the task's objective. Where order matters, express that order as a constraint rather than requiring the entire reference trace.
**How can evaluators handle borderline alternatives?**
Inspect whether the alternative changes commitments, exposes information or leaves additional work unresolved. Two traces that end with the same status label may still have different business effects. Conversely, different intermediate statuses may be harmless.
A task pack could include accepted outcome predicates and explicit prohibited transitions. Keep a reference trace as an explanation and debugging aid. This approach reduces false failures caused by stylistic differences while retaining strict checks on the decisions and state changes that matter.
Score business invariants without forcing every valid solution into one reference trace.
- Audience:
- Evaluation teams
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- When should escalation count as a successful agent outcome?
- What should buyers inspect before trusting a generated handover sample?
- How can an evaluation test safe retries after an uncertain tool response?
- How can an evaluation expose decisions based on stale evidence?
- How should handover evaluation go beyond checking task completion?
- How can we tell whether an agent learned the handover rule rather than a routing shortcut?