How should an evaluation team scope a pilot for booking datasets and tasks?

Start from what you need to learn, not from volume. Three or four evaluation questions are enough, such as: can the agent read open conditions, does it hold files that should not be booked, and does it reconcile schedules for non-standard products?

Then scope the pilot around them.

**Product coverage.** Pick one or two products, for example term loans and asset finance, rather than every product at once.

**Exception families.** Choose the failure patterns that matter to you, such as open conditions, missing approver references and settlement mismatches.

**Output shape.** Decide on formats (JSONL or Parquet), whether you want a static dataset, a task pack with expected snapshots, or a simulated environment with tool actions.

**Grading approach.** State how you will judge results, including whether process rules and reason matching apply.

**Limits.** Be clear about intended use, such as evaluation only or also training, and about holdout handling. Licence terms, quantities and timelines are agreed in the scoping conversation, not assumed.

Finally, plan a review step. Inspect the sample for consistency, label quality and grounding, then decide what to widen. Treat the pilot as a way to test fit, not as a final specification.