How can an evaluation team inspect the quality of generated booking tasks before using them?
Quality inspection covers internal consistency, policy alignment, label correctness, difficulty spread and grounding review, each of which can be tested on a sample before committing.
Inspect a sample in layers, starting with the cheapest checks.
**Structural consistency.** Do foreign keys resolve? Does every schedule reconcile to its facility terms, including residual lines? Do status histories follow allowed transitions?
**Policy alignment.** Apply the generated policies to each task by script and confirm the labelled disposition. A file marked `book` should pass every rule; a file marked `hold` should fail at least one, and the stated blocker should be the real one.
**Label correctness.** Have a domain reviewer, such as someone from credit operations, read a small set of tasks cold and write down what they would do. Compare to the labels and discuss disagreements.
**Difficulty spread.** Check that tasks are not all clean bookings or all traps. Tags for difficulty let you report results by slice.
**Leak checks.** Make sure the expected snapshot, labels and reasons are not visible in the agent's inputs.
**Grounding and disclosure review.** The data is generated, but ask how it was grounded and what overlap checks were run. Review a sample for anything that looks like it could identify a real organisation or person.
None of this proves the data is a perfect mirror of any one business. It shows whether it is coherent enough to grade against.
Run structural, policy and label checks on a sample first; they reveal most problems before any agent is scored against the data.
- Audience:
- Evaluation teams
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- How do I know whether my success criteria for booking tasks are strict enough?