What should I ask for in a sample or pilot before committing to a larger release?
A checklist of sample contents and terms to request, covering linked records, hard-case tasks, readable graders, reset, generation method and licence scope.
Before committing to a larger release, ask for a sample or pilot that lets you test each of the points below.
- A small set of linked records covering bank lines, open items, policies and cases, with a loader that checks keys and sums. - Tasks that include combined remittances, short-pays, duplicates and at least one case where the correct outcome is escalation. - Expected outcomes expressed as rules or acceptable sets, not only a single answer per task. - Graders delivered as readable code, with per-dimension scores and named critical failures. - A reset mechanism that returns the environment to named snapshots, with an action log you can replay. - A written description of how the data was generated and how disclosure risk was reviewed. - Written licence terms covering training, evaluation and held-out use, and what you may share with others. - A route to report tasks you consider ambiguous or wrong, and a stated way corrections reach later versions.
A sample that includes tasks, graders, reset and written terms lets you test fit with your agent before any larger commitment.
- Audience:
- Agent developers
- Content type:
- Checklist
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What delivery formats, splits and versioning should I agree before a first sample?
- What does a scoping meeting for a freight billing agent pilot produce?
- Will I receive real customer payments, and does generated data settle privacy questions?
- How should I scope a first engagement for a payment matching agent?
- What does a scoping conversation look like for an insurer premium desk pilot?
- What licence limits and permitted uses should I ask about?