How can a lab test whether a model learned the rule rather than the wording?
Controlled case variants can separate sensitivity to decision-relevant facts from reliance on familiar wording, record identifiers or repeated narrative patterns.
**What makes a useful pair of evaluation cases?**
Keep the business structure stable while changing one fact that should alter the decision. A task can change whether the requester is also the required independent approver, while leaving the amount, evidence and proposed correction unchanged. The expected outcome changes because the independence condition changes.
**Should every variation change the answer?**
No. Other pairs can alter irrelevant wording, record identifiers or document order while preserving the valid outcome. Together, the pairs test both appropriate sensitivity and resistance to superficial variation.
**How should training and evaluation material be separated?**
Ask for case-family identifiers and generation lineage suitable for grouping related variants. A split by individual rows can place closely related cases on both sides, making performance harder to interpret.
**What should the buyer review?**
Inspect the stated causal difference for each decision-changing pair and confirm that no unintended field reveals the expected answer. A model's result is more informative when the pack explains exactly which business fact should drive the change in behaviour.
Use paired cases to inspect whether model decisions change for the right reasons.
- Audience:
- Model labs
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can developers evaluate a correction beyond checking the final balance?
- What should buyers inspect before trusting a generated correction pack?
- How should an agent behave when a posting request times out?
- How do we distinguish a justified pause from an agent avoiding the task?
- How should evaluation hooks avoid leaking the expected correction to the agent?
- Which hooks reveal duplicate posting after a tool timeout?