What checks should we run on a sample before accepting it?
A checklist of hand checks, baseline tests and replay tests a reseller's reviewer can run on a pilot sample before deciding whether to list it.
Run these checks on a pilot sample before deciding whether it is fit to list.
- **Hand solve:** pick a spread of cases across causes and domains and solve each from the visible evidence alone. - **Gap arithmetic:** confirm that the difference between the records equals what the stated cause would produce. - **Policy permission:** check that every expected action is allowed by the attached policy and the acting role's permissions. - **Leakage scan:** look for answer clues in field names, record ordering, case titles and note wording. - **Stop cases:** confirm that some cases correctly require escalation or no change, and that they are labelled. - **Baseline test:** run a simple rule, such as matching on amount, and see which cases it solves; those cases may be too easy. - **Replay test:** run a case in the environment twice from reset and confirm the starting state is identical. - **Disclosure note:** record any grounding or disclosure concern and raise it before onward licensing.
Solving cases by hand, testing for leakage and replaying from reset reveals defects that would otherwise reach your customers after listing.
- Audience:
- Licensed resellers
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How does path-level scoring separate two agents that both match the totals?
- How can a buyer inspect the quality of generated mismatch cases?
- What checks should I run on a pack before using it for evaluation?
- How should I score an agent that reaches the right fix for the wrong reason?
- Where do reset, trace and grading hooks fit around an agent run?
- How do we know a generated mismatch case is solvable from what the agent can see?