Skip to content
MirrorFirm

How can we inspect the quality of generated mismatch cases before using them?

Sets out the inspection steps a buyer can run on a sample: solvability, internal consistency, leakage, realism of process and disclosure risk.

Quality can be inspected on a sample before any larger commitment. Five checks cover most of the risk.

**Solvability.** Give a reviewer the case and records, without the answer key, and ask them to find the cause. If an experienced reviewer cannot, the case is underspecified.

**Internal consistency.** Totals, dates and links should agree across files. A payment that settles before it was created, or a link to a missing record, signals a generation fault.

**Leakage.** Search case titles, notes and field names for words that give away the cause. Check whether a model with no tool access can guess the answer from the text alone.

**Process realism.** Ask someone who does this work whether the approvals, handovers and reason codes resemble their own. Differences are not always faults, but they should be understood.

**Disclosure and grounding.** The output is generated, but a buyer should check whether any identifiers, names or patterns look like real entities, and ask how the generation was grounded.

Record the results per case type, not as a single score. A release can be strong on duplicates and weak on cut-off timing, and you want to know that before you build a benchmark on it.

A short inspection routine on a sample shows whether cases are solvable, consistent, free of leaks and close enough to your process to be useful.

Audience:
Evaluation teams
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Reconciliation mismatch