How can a buyer inspect the quality of generated schedule data before using it?
An explanation of structural, semantic and behavioural checks a buyer can run on a sample, from key integrity and arithmetic through to policy fidelity and review of disclosure risk.
Inspection works best in layers, starting with what a script can check and ending with what needs a domain reader.
**Structural checks.** Do all foreign keys resolve? Do instalments sum to the agreement total after each adjustment? Are dates ordered and do superseded schedule versions form an unbroken chain? These are mechanical and should pass in full before anything else is reviewed.
**Semantic checks.** Does each adjustment cite a reason, an approver with the right role and a policy version that was in force on that date? Do notices describe the schedule version they claim to?
**Behavioural checks.** Run a baseline model or a simple scripted agent through a few tasks. Tasks that every attempt passes or every attempt fails deserve a look, since they may be too easy or have a faulty expected answer.
**Domain review.** Ask someone who has worked in the relevant operation to read a handful of episodes and say whether the handovers, escalations and exceptions read as plausible.
**Grounding and disclosure.** The material is generated, informed by how businesses operate. A buyer should still check that no text resembles a specific real organisation or person, and that the sample does not suggest more than the generation process can support.
Record what you checked and what you found, so that a pilot can address specific gaps.
Quality can be tested in layers, from mechanical integrity through policy fidelity, before any of the material reaches a training run.
- Audience:
- Model labs
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?