Skip to content
MirrorFirm

How can an evaluation team inspect the quality of generated booking tasks before using them?

Quality inspection covers internal consistency, policy alignment, label correctness, difficulty spread and grounding review, each of which can be tested on a sample before committing.

Inspect a sample in layers, starting with the cheapest checks.

**Structural consistency.** Do foreign keys resolve? Does every schedule reconcile to its facility terms, including residual lines? Do status histories follow allowed transitions?

**Policy alignment.** Apply the generated policies to each task by script and confirm the labelled disposition. A file marked `book` should pass every rule; a file marked `hold` should fail at least one, and the stated blocker should be the real one.

**Label correctness.** Have a domain reviewer, such as someone from credit operations, read a small set of tasks cold and write down what they would do. Compare to the labels and discuss disagreements.

**Difficulty spread.** Check that tasks are not all clean bookings or all traps. Tags for difficulty let you report results by slice.

**Leak checks.** Make sure the expected snapshot, labels and reasons are not visible in the agent's inputs.

**Grounding and disclosure review.** The data is generated, but ask how it was grounded and what overlap checks were run. Review a sample for anything that looks like it could identify a real organisation or person.

None of this proves the data is a perfect mirror of any one business. It shows whether it is coherent enough to grade against.

Run structural, policy and label checks on a sample first; they reveal most problems before any agent is scored against the data.

Audience:
Evaluation teams
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Loan booking