What should an environment builder request in a pilot sample?
A checklist of items to request in a pilot sample so it can be loaded, run against a reference agent and judged against your own criteria.
Ask for these items so the sample can be loaded, run and judged on your own terms.
- **Schema and data dictionary** covering every table, field type, key and enumerated value, with a note on which fields are rule-derived and which are free text. - **Boundary-heavy case mix** with cases below, at and above each authority limit, plus within-authority cases to test over-escalation. - **Task file** with start state, goal, permitted tools, expected end state and failure-mode labels per task. - **Senior response policies** that can be switched per task, including approve, amend, decline and return-for-information. - **A reference solution** for each task, even if scripted, so you can confirm solvability. - **Seed and generation notes** stating how the sample can be regenerated or varied, and which parameters you can change. - **Grounding and disclosure notes** describing what the generation was informed by at a general level and what checks you should run yourself. - **A change log format** so feedback from your review can be tracked into the next iteration.
Be explicit about your tool names and error types in the request. Agree in advance how you will judge the sample, and what result would lead you to widen or stop the pilot.
A sample that includes schemas, labels, reference solutions and a change log can be tested in days of your own effort, not weeks of guesswork.
- Audience:
- Environment builders
- Content type:
- Checklist
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What licensing questions should I raise for generated escalation material?
- What might a first escalation pilot show an agent team?
- What should my team prepare to start an escalation pilot?
- How should I scope a first escalation pilot with MirrorFirm?
- What should I ask for in an escalation sample or pilot?
- What licensing, limits and boundaries should a buyer ask about before a pilot?