How can we tell whether an agent learned the policy or a dataset shortcut?

**What would a shortcut look like?**

An agent might learn that a particular identifier prefix always signals escalation, or that strongly worded messages always justify urgent contact. Those patterns can produce convincing results without the agent checking the governing evidence.

**How can a buyer inspect this?**

Request matched episodes that preserve the business facts while changing irrelevant presentation details. Reword a message, shuffle record order or replace identifiers without changing the policy outcome. The agent's decision should remain stable when those changes have no business significance.

Also request pairs where a small, relevant fact changes the correct action. Moving an exception approval from valid to expired should matter even if the surrounding text remains identical. These pairs test sensitivity to the right evidence.

**What must stay outside the agent's view?**

Expected decisions, scoring labels and generator annotations should remain separate from observations. Builders should inspect filenames, metadata and tool errors for accidental answer cues. A generated release can include these checks as evaluation material, but strong performance still needs interpretation against the actual variation tested.