Skip to content
MirrorFirm

How can buyers test whether an agent understands referral triggers rather than keywords?

Paired cases can vary one decision-relevant fact while holding other details steady, revealing whether referral behaviour responds to policy evidence or superficial language.

A useful evaluation includes pairs of cases that differ in one fact relevant to the supplied referral policy. One customer requests a payment-date change for convenience. Another makes the same request but says they cannot meet essential expenses. The surrounding account history and wording can remain closely matched.

The expected behaviour should follow the policy distinction, not a word such as payment or struggling. Other pairs can keep the underlying difficulty constant while changing phrasing, channel or irrelevant account details. These comparisons test whether the agent's decision remains stable when it should.

Score the decision and its supporting evidence separately. An agent can select the correct route for an unsupported reason, which may fail when the next case changes slightly. Conversely, an agent may correctly recognise missing information and ask a necessary question before choosing a route.

MirrorFirm could generate these controlled comparisons with explicit changed-fact annotations for evaluators. Keep those annotations outside the agent's input. Buyers can inspect whether each pair has a defensible expected distinction and whether the evaluation rewards appropriate clarification rather than forcing premature certainty.

Use controlled comparisons to expose brittle recognition and unjustified referrals.

Audience:
Agent developers
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Hardship referral