Skip to content
MirrorFirm

What does a sensible two-stage sample review look like for a lab?

An applied research team reviews a small generated sample on structure and policy first, then runs a model against a slice, and only then decides whether to widen scope.

An applied research lead at a model lab wants to know whether set-up material suits an agent evaluation. They ask MirrorFirm for a sample covering insurance policy issuance: a bound quote becomes a policy record, a billing schedule and a certificate request.

**Stage one: inspect.** An engineer runs the validation script. Every `quote_id`, `policy_id` and `billing_id` resolves, and statuses follow the permitted transitions. A reviewer with insurance operations experience walks through a handful of tasks with the written underwriting rule. They find one task where the rule is ambiguous and log it as `REV-014`. The privacy lead searches the sample for identifiers and unusual free-text detail, and records the result against the sample reference.

**Stage two: run.** The team loads a slice into their harness, resets the environment per task and runs a current model. They tabulate failures by mode and notice the model often issues the certificate before the first billing record exists.

**Decision.** Because the findings are specific, they ask for a revised sample that fixes `REV-014` and adds more tasks that probe sequencing. They do not yet request volume.

**What it teaches:** a staged review produces concrete feedback to the supplier and a clear basis for the next step. All of the material is generated, so the grounding and disclosure review belongs in stage one.

Staging the review keeps early effort cheap and ties any expansion to evidence from your own models and your own reviewers.

Audience:
Model labs
Content type:
Scenario
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Account setup