How can evaluators detect shortcuts and answer leakage?

**What counts as answer leakage here?**

A visible field such as `requires_new_policy: true` can reveal the central judgement the task is intended to test. Filenames, record ordering and convenient wording patterns can also disclose expected actions without a deliberate answer field.

**How can a buyer inspect this?**

Review the agent-visible package separately from evaluator metadata. Look for labels, comments and joinable tables that expose expected outcomes. Check whether record identifiers or document lengths systematically distinguish affected work from unaffected work.

**What practical test helps?**

Use matched cases with the same layout and similar language but different decisive evidence. A generated pair can differ only in the recorded acceptance event that determines applicability. If the agent produces the same action for both, it may be ignoring the relevant condition.

**Should every structured hint be removed?**

No. Business systems legitimately contain derived fields. The task specification should state whether validating such a field is part of the assignment or whether the field is trusted input. The key is to align available information with the reasoning capability being measured and document any shortcuts that remain possible.