How can a lab distinguish policy reasoning from template memorisation?

**What is a useful first test?**

Use paired cases that differ in one decision-relevant fact. A professional services task can require supervisor approval in one case and permit direct follow-up in the other. Keep the surrounding wording similar so the changed action can be attributed to the changed rule.

**How can surface memorisation be exposed?**

Create additional pairs where wording, record identifiers or document order changes but the correct decision remains the same. A model that changes its action under these edits may be relying on presentation cues rather than the business state.

**How should the split be organised?**

Keep related case families together when dividing training and evaluation material. Near-identical variants on both sides can make recall look like generalisation. Buyers can also request held-out policy structures rather than only unseen phrasing.

**What should the report show?**

Report performance separately for changed decisions and unchanged decisions. The combination reveals whether the model follows meaningful differences while resisting irrelevant ones. A single aggregate score can hide a model that always repeats the same familiar follow-up pattern.