How should an evaluation team scope a pilot for approval-authority tasks?
Explains how to define domain, roles, limit types, conditions and format for a pilot, and what a sensible first step looks like before committing to a larger release.
Scope the pilot around the decision you want to test, not around volume.
**Choose one domain.** Pick a setting your agents will meet, such as procurement, claims payment, credit variation or refunds. Mixing domains early makes it hard to tell whether a failure comes from the model or the setting.
**Define the authority model.** State which limit types matter: single-item amount, aggregate exposure, risk band, category or role. Decide whether delegations, acting arrangements and revocations are in scope.
**List the conditions to test.** Choose from boundary amounts, linked requests, expired grants, self-approval and conflicts. Ask for each to be tagged so you can report per condition.
**Pick the form.** A task pack is the simplest start: prompts, records and gold outcomes. A simulated environment adds tools, state and reset, and takes more scoping.
**Set your format.** Say whether you need JSONL, Parquet or both, and share your harness conventions so field names fit.
**Agree how you will judge it.** Write down the checks and success criteria before you see the sample.
All content is generated, so ask how it was grounded and plan a disclosure review. Licence terms, quantities and timing are agreed with MirrorFirm during scoping, and this description does not set them.
A narrow pilot with explicit domain, limit types and failure conditions lets you test fit and quality before agreeing a larger scope.
- Audience:
- Evaluation teams
- Content type:
- Explanation
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- How does reviewing a sample change the scope of a freight claims pilot?
- What limits apply when licensing generated approval datasets or environments?
- How do I load a first approval sample into my agent harness?
- How should we scope a first pilot for approval-handling agents?
- What should I ask for in a sample or pilot of approval material?
- What does a scoping session for an approvals pilot produce?