How do I inspect the quality of generated booking records?
Inspect consistency, behaviour that matches the work, diversity and disclosure risk, then decide whether cases are for training, evaluation or both and keep those sets separate.
Inspecting generated booking material means checking four things: whether it is internally consistent, whether it behaves like the work, whether it is varied, and whether it carries disclosure risk.
**Internal consistency.** Keys should resolve, amounts should reconcile, and dates should run in a sensible order. A repayment schedule should match the facility's limit, rate and term. A facility should not become ACTIVE with a condition still open unless an override is recorded.
**Behaviour like the work.** Look at the handovers. Do queries go from loan operations to credit and come back with answers? Are exceptions handled by role, with delegations respected? Ask a practitioner in lending operations to read a handful of cases and mark anything no operations team would do.
**Diversity.** Check that cases differ in product, size, exception type and outcome, rather than being the same case with new identifiers.
**Disclosure and grounding risk.** The output is generated, but that does not remove the need to look. Search for text that reads like a specific institution's documents, identifiers that resemble real ones, and passages that repeat across records. Treat this as an inspection step, not an assumption.
Finally, decide whether the release is for training, evaluation or both, and keep evaluation cases separate from training cases.
A short, repeatable inspection routine lets you judge a release on evidence and keep the same standard across every version you list.
- Audience:
- Licensed resellers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- How do I know whether my success criteria for booking tasks are strict enough?