Which quality checks reveal shortcuts that could undermine an evidence dataset?

Inspect the generated material for shortcuts that let an agent predict the expected finding without performing the evidence reasoning the task is intended to test.

- Check whether filenames reveal outcomes, such as every deficient case containing an attachment title that directly announces the missing control. - Compare record length across outcome classes, since consistently shorter incomplete cases can make document volume an unintended predictor of correctness. - Inspect identifier patterns for outcome encoding, including prefixes or ordered sequences that separate supported cases from cases requiring further investigation. - Review language diversity across roles so a particular writing style does not consistently identify the unreliable source or the correct conclusion. - Look for repeated distinctive passages across training and evaluation splits that would allow memorised associations to substitute for analysing new records. - Verify that difficult cases remain logically resolvable or explicitly unresolved, rather than becoming difficult through accidental contradictions in the generated material. - Test whether removing irrelevant formatting changes performance, which can reveal reliance on presentation cues rather than the relationships among supporting records.