What does a sensible two-stage sample review look like for a lab?
An applied research team reviews a small generated sample on structure and policy first, then runs a model against a slice, and only then decides whether to widen scope.
An applied research lead at a model lab wants to know whether set-up material suits an agent evaluation. They ask MirrorFirm for a sample covering insurance policy issuance: a bound quote becomes a policy record, a billing schedule and a certificate request.
**Stage one: inspect.** An engineer runs the validation script. Every `quote_id`, `policy_id` and `billing_id` resolves, and statuses follow the permitted transitions. A reviewer with insurance operations experience walks through a handful of tasks with the written underwriting rule. They find one task where the rule is ambiguous and log it as `REV-014`. The privacy lead searches the sample for identifiers and unusual free-text detail, and records the result against the sample reference.
**Stage two: run.** The team loads a slice into their harness, resets the environment per task and runs a current model. They tabulate failures by mode and notice the model often issues the certificate before the first billing record exists.
**Decision.** Because the findings are specific, they ask for a revised sample that fixes `REV-014` and adds more tasks that probe sequencing. They do not yet request volume.
**What it teaches:** a staged review produces concrete feedback to the supplier and a clear basis for the next step. All of the material is generated, so the grounding and disclosure review belongs in stage one.
Staging the review keeps early effort cheap and ties any expansion to evidence from your own models and your own reviewers.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot look like for a client and matter opening agent?
- What licence and usage limits should we ask about before starting?
- How should we scope a first pilot for set-up agents?
- What should we ask for in a sample or pilot release for set-up?
- How do we wire a pilot release into our agent harness on day one?
- How should delivery formats, versioning and data splits be agreed for a pilot?