Which checks reveal whether an agent understands applicability rather than memorising cues?

Use this checklist to build paired evaluations where each variation has a defined expected effect.

- Rename identifiers without changing the answer. - Paraphrase a clause without changing its meaning. - Reorder retrieved documents without changing authority. - Move a governing date across the applicability boundary. - Change a case attribute that activates a requirement. - Add irrelevant notes that should not affect action.

The pairs should isolate the changed factor. If a date-boundary test also changes the case category and approval state, a different result will not reveal which fact the agent used. Evaluator notes should state whether the expected decision stays the same or changes, and why.

For insurance tasks, a claim's submission channel might be irrelevant while its coverage category determines whether revised evidence is required. An agent that changes its decision when only the channel label changes may be relying on an accidental association.

Keep related variants together when separating training and evaluation material to reduce answer leakage. A buyer can ask for held-out policy structures as well as new wording, making the test more demanding than recognising familiar sentences.