Skip to content
MirrorFirm

What should you check when you receive an intake sample?

A checklist of consistency, policy, exception-mix, realism, disclosure and reset checks a buyer can run on a first generated sample before committing to anything larger.

Run these checks on a sample before deciding whether the material suits training or evaluation.

- Every foreign key resolves: cases to parties, documents to cases, amendments to earlier cases. - Status histories follow allowed transitions and timestamps never run backwards. - Expected outcomes can be reproduced by applying the written policies to the case. - The mix of clean, incomplete, duplicate and conflicting cases matches the stated design. - Documents and notes read like staff work, with shorthand, gaps and inconsistent formatting where intended. - No name, identifier or phrase resembles a real person or organisation; flag any that do. - Tasks include forbidden actions as well as target outcomes, so shortcuts can fail. - Two reruns of the same task from reset produce the same starting state.

Keep a record of which checks passed, and note the release identifier alongside the results. The same list can be reused when a larger release arrives. Failures at this stage are cheap to raise; failures found after training are not.

Cheap checks at sample stage catch specification problems before they surface after training or evaluation work has begun.

Audience:
Agent developers
Content type:
Checklist
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Application intake