Skip to content
MirrorFirm

How should a benchmark score several valid ways to complete a referral?

Outcome constraints and action dependencies can support fair scoring when different conversation sequences satisfy the same policy and reach an acceptable referral state.

**Must an agent match a reference conversation?**

No. A reference can demonstrate one acceptable path, but exact wording and turn order are often poor substitutes for operational correctness. An agent might confirm a contact preference before explaining the referral, while another does those steps in the opposite order.

**What should determine success?**

Define required outcomes, prohibited actions and genuine ordering constraints. If the supplied policy requires permission before submission, that dependency matters. If two information-gathering steps can occur in either order, the scorer should allow both.

**How should partial completion be handled?**

Record which requirements were satisfied and why progress stopped. A customer declining the offered referral can be an acceptable terminal outcome when the agent follows policy. It should not be scored as equivalent to an agent forgetting to submit an agreed request.

**Where does human review help?**

Review ambiguous explanations and cases where structured outcomes conflict with conversational evidence. Generated scoring specifications should make these disagreements inspectable. This gives evaluation teams a defensible basis for scoring alternative paths without turning every difference from the reference into a failure.

Avoid penalising valid approaches merely because they differ from a reference transcript.

Audience:
Evaluation teams
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Hardship referral