Skip to content
MirrorFirm

What should an environment builder request in a pilot sample?

A checklist of items to request in a pilot sample so it can be loaded, run against a reference agent and judged against your own criteria.

Ask for these items so the sample can be loaded, run and judged on your own terms.

- **Schema and data dictionary** covering every table, field type, key and enumerated value, with a note on which fields are rule-derived and which are free text. - **Boundary-heavy case mix** with cases below, at and above each authority limit, plus within-authority cases to test over-escalation. - **Task file** with start state, goal, permitted tools, expected end state and failure-mode labels per task. - **Senior response policies** that can be switched per task, including approve, amend, decline and return-for-information. - **A reference solution** for each task, even if scripted, so you can confirm solvability. - **Seed and generation notes** stating how the sample can be regenerated or varied, and which parameters you can change. - **Grounding and disclosure notes** describing what the generation was informed by at a general level and what checks you should run yourself. - **A change log format** so feedback from your review can be tracked into the next iteration.

Be explicit about your tool names and error types in the request. Agree in advance how you will judge the sample, and what result would lead you to widen or stop the pilot.

A sample that includes schemas, labels, reference solutions and a change log can be tested in days of your own effort, not weeks of guesswork.

Audience:
Environment builders
Content type:
Checklist
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Credit escalation