What checks should we run on a set-up sample?
A checklist of seven validation checks, from key resolution and condition carry-through to split hygiene and disclosure review, to run and record before a sample is listed or shared.
Run these checks on any sample before it is listed or passed to a customer, and record the result of each.
- **Key resolution:** every foreign key in accounts, services, tasks and events resolves to an existing parent record, and no table contains orphans or circular references. - **Condition carry-through:** each condition on an approval appears as a task, limit or flag on the resulting account, or is explicitly waived by an allowed role. - **Time order:** approvals precede accounts, accounts precede activation, and no event is stamped earlier than the record it changes. - **Exception variety:** duplicate, missing-evidence and amended-request cases differ in structure, not only in identifiers or wording. - **Scoring clarity:** each task has separate named checks, so a failed run identifies the behaviour that failed rather than returning one verdict. - **Split hygiene:** training and evaluation cases are marked, and no case appears in both. - **Disclosure review:** generated values are inspected for resemblance to real identifiers, rare combinations and source-specific details.
A fixed set of recorded checks makes sample reviews repeatable and gives you evidence to share with customers and with MirrorFirm.
- Audience:
- Licensed resellers
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?