How do I judge the quality of generated mismatch cases?

Quality in a mismatch dataset comes down to whether each case can be solved from the evidence provided, and whether it can be solved only by doing the work.

**Solvability.** A reviewer should be able to follow the evidence from the disagreement to the labelled root cause without outside knowledge. Cases meant to be unresolvable should say so in their metadata.

**Decoys.** Plausible but wrong explanations should fail on inspection. If a shortcut such as always trusting the later-dated record gives the right answer across most cases, the set is too easy to game.

**Consistency.** Amounts, dates, identifiers and totals should agree wherever they are meant to, and differ only where the case intends. Automated checks can test balances, link integrity and date order.

**Policy fit.** The expected outcome should follow from the stated policy, including cases where the policy says to escalate.

**Realism and disclosure.** Because the output is generated, check how closely it follows plausible operating practice, and look for details that seem to echo a specific organisation or person. Generation does not guarantee that such resemblance is absent.

A useful habit is to take a small sample, solve several cases by hand, and compare your reasoning with the labels before running any model on it.