Skip to content
MirrorFirm

What should we ask for in a first sample of booking data or tasks?

A checklist of items to request in an initial sample so it can be inspected for structure, labels, difficulty, grounding and fit with your evaluation harness.

Request a small sample that you can run your own checks on, and ask that it include the following.

- A manifest listing files, keys, schema version and generation seed. - Linked approvals, conditions, facilities, schedules and security items, with working foreign keys. - A task file with start and expected snapshots, expected dispositions and blocking items. - A mix of clean bookings, held files and returned files, with difficulty tags. - The policy set the labels were derived from, in a readable form. - A short description of how the content was generated and grounded, and what overlap checks were run. - A sketch of the tool actions and reset behaviour, if you want an environment. - A note on intended use, holdout handling and any limits on sharing.

On your side, decide in advance how you will review it: script checks for keys and schedule reconciliation, a domain reviewer reading a few tasks cold, and a quick run with a baseline agent to see whether the tasks discriminate. Write down what would make you widen, change or stop the pilot.

A well-specified first sample lets you test fit and quality quickly, and gives both sides a concrete basis for scoping a pilot.

Audience:
Evaluation teams
Content type:
Checklist
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Loan booking