How can a buyer inspect the quality of generated mismatch cases?
Describes layered inspection of generated cases: structure, cause fit, policy fit, variety, and grounding and disclosure review, followed by reading agent traces rather than only scores.
Quality here means the cases behave like reconciliation work, not that the fields look tidy. A buyer can inspect it in layers.
**Structure.** Check that identifiers resolve, timestamps are in a sensible order, amounts add up where they should, and the variance on each case equals the difference between the records.
**Cause fit.** Take a sample of cases and try to find the root cause by hand using only what an agent would see. If the cause is unreachable, the case is unfair. If it is visible in the first record, it is too easy.
**Policy fit.** Confirm each case cites a policy that decides which actions are allowed, and that the labelled acceptable actions follow from that text.
**Variety.** Look for different causes, different systems, noise records that are irrelevant, and some cases that should end in escalation.
**Grounding and disclosure.** The output is generated, so ask how it was shaped and inspect it for plausibility against your own process. Check for anything that looks like an identifiable real record. Generation does not remove the need for that review.
Then run your agent and read the traces, not just the scores. Where it fails tells you whether the data or the agent is the weak part.
Quality means cases behave like investigative work with a reachable cause, so test that by hand before you score any agent.
- Audience:
- Agent developers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How does path-level scoring separate two agents that both match the totals?
- What checks should I run on a pack before using it for evaluation?
- How should I score an agent that reaches the right fix for the wrong reason?
- Where do reset, trace and grading hooks fit around an agent run?
- How do we know a generated mismatch case is solvable from what the agent can see?
- What evaluation hooks should a mismatch environment expose for reset, traces and scoring?