Skip to content
MirrorFirm

How can a buyer inspect the quality of generated set-up data?

Quality can be inspected through referential integrity, policy consistency, exception coverage and a review by someone who has done the work, before any model is scored on it.

Inspect the data first, because a flawed case measures the case rather than the agent.

**Structural checks**

- Every foreign key resolves, and statuses follow allowed transitions. - Dates are coherent: effective dates do not precede approvals, and expiry dates fall where they should. - Each case has exactly one expected end state, and it can be derived from the inputs and policy.

**Policy checks**

- The written policy and its structured rule agree. - Where the policy is ambiguous on purpose, the case is tagged as such and the expected outcome is justified.

**Coverage checks**

- Count cases by exception type and by starting status to see whether any category is thin. - Look for repeated phrasing or identical structure that would let a model memorise a pattern.

**Practitioner review**

Ask someone who has done account or service set-up to read a sample of cases and mark anything that would not happen in their workplace. Their notes are often the most useful signal.

**Disclosure review**

The output is generated, not real business records, but check whether any content looks like it was copied from a source and whether grounding is as claimed. Nothing here removes the need to inspect it.

Inspect the data before you trust the scores: a flawed case measures the case, not the agent.

Audience:
Evaluation teams
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Account setup