Skip to content
MirrorFirm

How can a buyer inspect the quality of generated mismatch cases?

Describes layered inspection of generated cases: structure, cause fit, policy fit, variety, and grounding and disclosure review, followed by reading agent traces rather than only scores.

Quality here means the cases behave like reconciliation work, not that the fields look tidy. A buyer can inspect it in layers.

**Structure.** Check that identifiers resolve, timestamps are in a sensible order, amounts add up where they should, and the variance on each case equals the difference between the records.

**Cause fit.** Take a sample of cases and try to find the root cause by hand using only what an agent would see. If the cause is unreachable, the case is unfair. If it is visible in the first record, it is too easy.

**Policy fit.** Confirm each case cites a policy that decides which actions are allowed, and that the labelled acceptable actions follow from that text.

**Variety.** Look for different causes, different systems, noise records that are irrelevant, and some cases that should end in escalation.

**Grounding and disclosure.** The output is generated, so ask how it was shaped and inspect it for plausibility against your own process. Check for anything that looks like an identifiable real record. Generation does not remove the need for that review.

Then run your agent and read the traces, not just the scores. Where it fails tells you whether the data or the agent is the weak part.

Quality means cases behave like investigative work with a reachable cause, so test that by hand before you score any agent.

Audience:
Agent developers
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Reconciliation mismatch