How can we inspect the quality of generated mismatch cases before using them?
Sets out the inspection steps a buyer can run on a sample: solvability, internal consistency, leakage, realism of process and disclosure risk.
Quality can be inspected on a sample before any larger commitment. Five checks cover most of the risk.
**Solvability.** Give a reviewer the case and records, without the answer key, and ask them to find the cause. If an experienced reviewer cannot, the case is underspecified.
**Internal consistency.** Totals, dates and links should agree across files. A payment that settles before it was created, or a link to a missing record, signals a generation fault.
**Leakage.** Search case titles, notes and field names for words that give away the cause. Check whether a model with no tool access can guess the answer from the text alone.
**Process realism.** Ask someone who does this work whether the approvals, handovers and reason codes resemble their own. Differences are not always faults, but they should be understood.
**Disclosure and grounding.** The output is generated, but a buyer should check whether any identifiers, names or patterns look like real entities, and ask how the generation was grounded.
Record the results per case type, not as a single score. A release can be strong on duplicates and weak on cut-off timing, and you want to know that before you build a benchmark on it.
A short inspection routine on a sample shows whether cases are solvable, consistent, free of leaks and close enough to your process to be useful.
- Audience:
- Evaluation teams
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How does path-level scoring separate two agents that both match the totals?
- How can a buyer inspect the quality of generated mismatch cases?
- What checks should I run on a pack before using it for evaluation?
- How should I score an agent that reaches the right fix for the wrong reason?
- Where do reset, trace and grading hooks fit around an agent run?
- How do we know a generated mismatch case is solvable from what the agent can see?