How should evaluation handle more than one acceptable destination?
Routing quality can be evaluated against admissible outcomes and required evidence, avoiding a false failure when several destinations satisfy the same policy.
Some routing tasks permit several correct outcomes. A utility’s generated operating rules can allow either of two regional support queues to handle a general account enquiry, provided the selected queue covers the service location and accepts that request category.
An evaluator can express correctness as constraints over the final state. The destination must be eligible, the request must have an accountable owner and any required evidence must exist. A preferred destination can be scored separately if the policy describes a preference rather than an obligation.
The evaluation should also distinguish invalid routing from justified deferral. A request lacking the service location may belong in an authorised information-gathering state. Marking every non-assignment as failure would reward agents for guessing.
A buyer can ask MirrorFirm for generated tasks with admissible outcome sets and the policy reasons behind them. Inspect whether those sets include all supported alternatives and exclude destinations that merely have similar names. This helps builders avoid a brittle answer key and makes disagreements reviewable: the question becomes which constraint was violated, rather than whether the agent selected the evaluator’s favourite queue.
Score policy compliance and completed responsibility instead of matching one arbitrary queue identifier.
- Audience:
- Environment builders
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can buyers test whether an agent uses the right evidence to choose a destination?
- Which evaluation hooks reveal whether a routing action preserved the required handover?
- What should buyers inspect before using a generated routing corpus?
- How can buyers detect answer leakage in generated routing evaluations?
- How should evaluation handle a destination that changes while the agent is working?
- What should buyers check in generated content before trusting agent scores?