Skip to content
MirrorFirm

How can I inspect the quality of generated booking data before using it?

An explanation of the structural, behavioural and grounding checks a buyer can run on a generated sample before using it for training or evaluation.

Treat a sample as something to test, not to accept. Three kinds of check cover most of the risk.

**Structural checks**

Confirm every foreign key resolves, identifiers are unique, dates are ordered sensibly (approval before booking, booking before first instalment) and amounts reconcile between schedules and ledger entries. These can be scripted in an afternoon.

**Behavioural checks**

Replay a few tasks by hand, or with a scripted reference solver. Each task should be solvable from the information in the environment, and each trap should be detectable. If a reference solver cannot reach the expected state, the task or the data is at fault.

**Grounding and disclosure checks**

The output is generated, informed by direct observation of how real businesses operate. That does not remove the need to look. Review samples for implausible combinations, repeated templates and any text that appears to reproduce a real organisation or individual. Ask MirrorFirm how a release was produced and what review it went through.

**Distribution checks**

Look at the spread of terms, amounts, exception types and outcomes. A set where nearly every file books cleanly will not teach an agent to stop.

Write down your thresholds before you look, so the decision does not drift.

Quality is something you can test with queries and replays, not something to take on trust, and the checks are cheap to run on a sample.

Audience:
Environment builders
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Loan booking