How should evaluators check that agents cannot solve handovers through dataset shortcuts?

Apply these checks before interpreting benchmark results as evidence of handover capability.

- Check that record identifier patterns do not reveal the correct destination or outcome. - Remove evaluator labels from searchable attachments, filenames and agent-visible metadata. - Vary message phrasing independently of whether a transfer should succeed or fail. - Ensure document length does not consistently identify the authoritative source. - Keep closely related case variants together when assigning training and evaluation splits. - Swap irrelevant surface details and confirm that the expected decision remains unchanged. - Change one decisive condition and confirm that the expected decision changes accordingly.

These checks examine whether the task demands the intended reasoning. A model that predicts rejection from a recurring phrase has not necessarily learned to inspect missing dependencies. Likewise, success on a close rewrite of a training case says little about handling a new combination of conditions.

Retain an audit record of the cues inspected and the variants tested. When a shortcut is found, revise the generated material or narrow the interpretation of the result before comparing systems. The benchmark should reward attention to business evidence, not accidental regularities in its construction.