How can we tell whether agents learned the task or memorised dataset shortcuts?

**Is a random train-test split sufficient?**

It may leave closely related generated cases on both sides. If records share a template, exception pattern or generation seed family, an agent can exploit similarities without learning how to assemble evidence.

**What should be separated?**

Group related case variants before splitting. Hold out combinations of process conditions, such as delegated approval with a reopened request, rather than only holding out new record identifiers. Keep generation lineage metadata available to evaluators so they can inspect these boundaries.

**How do we detect superficial cues?**

Use paired checks. Change document formatting while preserving the evidence logic, then change the logic while preserving the formatting. A sound result should remain stable in the first case and respond appropriately in the second. Remove accidental cues such as failure cases always having longer identifiers.

**What belongs in the agent’s input?**

Only the material needed to perform the task. Answer keys, generation notes and split labels should remain evaluator-only. Report performance by held-out condition as well as overall outcome, so strong results on familiar patterns do not conceal failure on a new process combination.