How can a buyer inspect the quality of generated set-up data?
Quality can be inspected through referential integrity, policy consistency, exception coverage and a review by someone who has done the work, before any model is scored on it.
Inspect the data first, because a flawed case measures the case rather than the agent.
**Structural checks**
- Every foreign key resolves, and statuses follow allowed transitions. - Dates are coherent: effective dates do not precede approvals, and expiry dates fall where they should. - Each case has exactly one expected end state, and it can be derived from the inputs and policy.
**Policy checks**
- The written policy and its structured rule agree. - Where the policy is ambiguous on purpose, the case is tagged as such and the expected outcome is justified.
**Coverage checks**
- Count cases by exception type and by starting status to see whether any category is thin. - Look for repeated phrasing or identical structure that would let a model memorise a pattern.
**Practitioner review**
Ask someone who has done account or service set-up to read a sample of cases and mark anything that would not happen in their workplace. Their notes are often the most useful signal.
**Disclosure review**
The output is generated, not real business records, but check whether any content looks like it was copied from a source and whether grounding is as claimed. Nothing here removes the need to inspect it.
Inspect the data before you trust the scores: a flawed case measures the case, not the agent.
- Audience:
- Evaluation teams
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?