Skip to content
MirrorFirm

How can a lab test whether escalation decisions respond to the right facts?

Counterfactual case pairs can isolate the fact that should change an escalation decision while keeping unrelated wording and background details stable.

A model can select the expected escalation action for the wrong reason. To inspect that risk, a buyer can ask for paired cases that differ in one decision-relevant fact while preserving the rest of the complaint.

In one professional services case, the reviewer is independent of the disputed work. In its paired case, the proposed reviewer approved that work. If the policy requires independent review, this change should alter the routing decision. Changing the customer's writing style should not have the same effect.

Evaluation can examine both directions. The model should change its action when a decisive fact changes, and retain its action when an irrelevant detail changes. Explanations should identify the changed fact and connect it to the applicable rule.

MirrorFirm could generate controlled variations with declared intervention fields and expected behavioural differences. Reviewers should confirm that the intervention has not accidentally changed another material condition. These tests are useful for training contrastive judgement and for diagnosing shortcut learning, because a high aggregate success rate can otherwise hide dependence on superficial language.

Measure sensitivity to decisive evidence and resistance to irrelevant variation through controlled case comparisons.

Audience:
Model labs
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Complaint escalation