Skip to content
MirrorFirm

How do we check that evaluation answers have not leaked into training material?

Review datasets need split boundaries that account for shared entities, repeated narratives and hidden solution artefacts, because different record identifiers alone do not establish independent evaluation cases.

**Are different review identifiers enough to separate training and evaluation?**

No. Two reviews can expose the same underlying account history or repeat the same decision rationale under different identifiers. A customer may effectively encounter the evaluation answer during training even when exact record matching finds nothing.

**What should we inspect?**

Check shared entities, copied narrative passages, repeated policy-specific answer patterns and references to solution files. Inspect retrieval indexes as well as the visible dataset. An evaluator-only rationale stored in a searchable document collection can undermine an otherwise careful split.

**Should every shared policy be removed?**

Not necessarily. If the task measures applying a known policy to new cases, sharing the policy is intentional. If it measures adaptation to unfamiliar rules, the policy family may also need separation. State which capability the split is intended to test.

**What changes when we combine releases?**

Resellers should reassess the combined corpus rather than relying solely on each component's original split labels. A case held out in one product may closely match training material in another. Ask for grouping metadata that supports checks across the products customers are likely to use together.

Resellers can inspect contamination risks before combining generated packs or distributing separate training and evaluation products.

Audience:
Licensed resellers
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Periodic review