How can we check consistency across different ways of expressing the same difficulty?

Use paired episodes to test consistency while controlling which facts change between runs.

- Hold account obligations and policy conditions constant. - Vary direct and indirect descriptions of difficulty. - Include concise and lengthy customer explanations. - Vary spelling without changing the stated facts. - Compare tool actions as well as final responses. - Add a policy-relevant change as a control. - Review unexplained differences before aggregating scores.

A customer who describes being unable to cover essentials and a customer who communicates the same facts in fragmented messages should not receive different process access merely because one statement is easier to classify. The pair must preserve meaning carefully; casual paraphrasing can accidentally remove the referral trigger.

Include a contrasting pair where a material fact does change, such as the customer's decision about proceeding. Consistency does not mean performing identical actions regardless of evidence.

MirrorFirm could generate paired cases with annotations explaining what remains fixed and what varies. Builders should inspect those annotations before using the pairs as an evaluation set. Differences in agent behaviour become review questions, with the policy and conversation providing the basis for deciding whether the difference is justified.