What does reviewing a pilot sample and revising the scope look like in practice?
An evaluation lead reviews a lending sample against written checks, finds senior responses too uniform, and revises the case mix before the scope is agreed.
An evaluation lead is building a benchmark for lending agents and asks MirrorFirm for a sample based on Aster Financial. The request is logged as **PILOT-014** with status `DRAFT_SCOPE`.
The lead's request names one product, unsecured business lending, two reviewer tiers above the credit officer, and a case mix that includes connected-exposure cases and within-authority controls. It also asks for the expected outcomes to be delivered as a separate file.
The sample arrives and the lead's team runs their mechanical checks. Exposure arithmetic and referential integrity pass. On reading the referral packages, the team notes that senior-reviewer responses are almost all approvals, so the return-for-information path is barely exercised. They record this against the sample and ask for a revised mix with more returns and at least one case where the second tier declines.
The lead updates `PILOT-014` to `SCOPE_AGREED`, with the changed mix, the evaluation-only use, and the acceptance checks written into the scope. Only then does the team ask for a larger set.
**What it teaches:** a sample is a way to find design gaps cheaply. Reviewing it against written checks, and recording what was wrong, turns the next request into a specific change instead of a general request for more data.
Reviewing a sample against written checks turns the next request into a specific change instead of a general call for more data.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What licensing questions should I raise for generated escalation material?
- What might a first escalation pilot show an agent team?
- What should my team prepare to start an escalation pilot?
- How should I scope a first escalation pilot with MirrorFirm?
- What should I ask for in an escalation sample or pilot?
- What licensing, limits and boundaries should a buyer ask about before a pilot?