What quality checks reveal shortcuts in a generated referral benchmark?

Apply this checklist before using a generated referral pack to compare agents or report evaluation results.

- Check identifiers for encoded outcome labels. - Inspect templates for answer-revealing phrases. - Keep related case families within one split. - Compare message length across outcome categories. - Review whether missing fields predict the answer. - Verify that policy retrieval requires meaningful selection. - Inspect easy cases alongside difficult exceptions.

An agent can exploit regularities that were never intended as evidence. If every referral-positive conversation is longer than every other conversation, a model may appear capable while relying on length. If related versions of the same case appear in development and held-out sets, reported generalisation becomes difficult to interpret.

Ask for construction metadata that supports this audit without placing answer labels in agent-visible records. Evaluate whether the metadata describes shared templates, case families and controlled variations clearly enough to support appropriate splitting.

The purpose is not to remove all regularity. Business records naturally contain repeated structures. The check is whether those structures provide the answer without requiring the policy application, evidence interpretation or action selection the benchmark claims to measure.