What should we ask for in a first sample of booking data or tasks?
A checklist of items to request in an initial sample so it can be inspected for structure, labels, difficulty, grounding and fit with your evaluation harness.
Request a small sample that you can run your own checks on, and ask that it include the following.
- A manifest listing files, keys, schema version and generation seed. - Linked approvals, conditions, facilities, schedules and security items, with working foreign keys. - A task file with start and expected snapshots, expected dispositions and blocking items. - A mix of clean bookings, held files and returned files, with difficulty tags. - The policy set the labels were derived from, in a readable form. - A short description of how the content was generated and grounded, and what overlap checks were run. - A sketch of the tool actions and reset behaviour, if you want an environment. - A note on intended use, holdout handling and any limits on sharing.
On your side, decide in advance how you will review it: script checks for keys and schedule reconciliation, a domain reviewer reading a few tasks cold, and a quick run with a baseline agent to see whether the tasks discriminate. Write down what would make you widen, change or stop the pilot.
A well-specified first sample lets you test fit and quality quickly, and gives both sides a concrete basis for scoping a pilot.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot look like for an agent team in vendor finance?
- What limits apply, and will we receive any real customer records?
- How could a pilot be delivered, versioned and wired into our CI?
- How should a team scope and license a generated booking pack?
- What should we ask for in a sample or pilot of generated booking material?
- How should a sample be delivered, versioned and load-checked in my pipeline?