How do I judge the quality of generated schedule data?
An explanation of the structural, behavioural and grounding checks a buyer can apply to generated schedule data before using it to train or evaluate an agent.
Quality has three layers, and each needs its own inspection.
**Structural quality.** Do the records agree with each other? Check that every foreign key resolves, that instalments sum to the contract amount, that superseded schedules point at a successor, and that dates fall on valid business days. These checks can be scripted and should pass in full.
**Behavioural quality.** Do the cases look like the work? Read a sample of cases end to end. Does the approval route match the policy? Do refusals carry sensible reasons? Do handovers include the information the next role would need? A reviewer with domain experience will spot unrealistic sequences faster than any script.
**Grounding and disclosure.** The data is generated, so ask how its patterns were informed by observed process behaviour, and what was done to avoid reproducing any real record. Inspect samples for unusual specificity, such as rare combinations of details that could point to a real case. Generation does not by itself guarantee privacy, so treat disclosure risk as something to test.
Then test usefulness. Run a baseline agent on a task sample and look at where it fails. If failures cluster on policy lookups and linked records, the pack is exercising the right skills. If every task is passed or failed uniformly, the difficulty is probably mis-set.
Check structure, behaviour and grounding separately, because data can be internally consistent and still unrepresentative of how a business operates.
- Audience:
- Agent developers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?
- Which failure modes should schedule tasks be designed to expose?