Skip to content
MirrorFirm

How can a lab test whether a model learned the rule rather than the wording?

Controlled case variants can separate sensitivity to decision-relevant facts from reliance on familiar wording, record identifiers or repeated narrative patterns.

**What makes a useful pair of evaluation cases?**

Keep the business structure stable while changing one fact that should alter the decision. A task can change whether the requester is also the required independent approver, while leaving the amount, evidence and proposed correction unchanged. The expected outcome changes because the independence condition changes.

**Should every variation change the answer?**

No. Other pairs can alter irrelevant wording, record identifiers or document order while preserving the valid outcome. Together, the pairs test both appropriate sensitivity and resistance to superficial variation.

**How should training and evaluation material be separated?**

Ask for case-family identifiers and generation lineage suitable for grouping related variants. A split by individual rows can place closely related cases on both sides, making performance harder to interpret.

**What should the buyer review?**

Inspect the stated causal difference for each decision-changing pair and confirm that no unintended field reveals the expected answer. A model's result is more informative when the pack explains exactly which business fact should drive the change in behaviour.

Use paired cases to inspect whether model decisions change for the right reasons.

Audience:
Model labs
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Ledger adjustment