Skip to content
MirrorFirm

What should I ask for in a sample or pilot before committing to a larger release?

A checklist of sample contents and terms to request, covering linked records, hard-case tasks, readable graders, reset, generation method and licence scope.

Before committing to a larger release, ask for a sample or pilot that lets you test each of the points below.

- A small set of linked records covering bank lines, open items, policies and cases, with a loader that checks keys and sums. - Tasks that include combined remittances, short-pays, duplicates and at least one case where the correct outcome is escalation. - Expected outcomes expressed as rules or acceptable sets, not only a single answer per task. - Graders delivered as readable code, with per-dimension scores and named critical failures. - A reset mechanism that returns the environment to named snapshots, with an action log you can replay. - A written description of how the data was generated and how disclosure risk was reviewed. - Written licence terms covering training, evaluation and held-out use, and what you may share with others. - A route to report tasks you consider ambiguous or wrong, and a stated way corrections reach later versions.

A sample that includes tasks, graders, reset and written terms lets you test fit with your agent before any larger commitment.

Audience:
Agent developers
Content type:
Checklist
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Payment allocation