Skip to content
MirrorFirm

How should an evaluation team scope a first mismatch pilot?

Recommends starting with one domain, a few cause types and a clear scoring design, then expanding based on what the sample shows.

Start narrow. A first pilot should answer one question: does generated material of this kind discriminate between the systems you care about, in your harness?

**Pick one domain and one pair of systems.** Bank feed against ledger, or claims against payments, is enough. Breadth can follow.

**Choose a small set of cause types.** Include at least one that is easy to mistake for another, such as a timing difference that explains only part of a gap.

**Decide the form.** Static cases suit offline scoring of written answers. A stateful environment suits tool-using agents and action scoring. A pilot can begin with static files and add an environment if the early results justify it.

**State your scoring design.** Say which labels you need, which violations disqualify a run and whether you will use a model grader.

**Agree inspection criteria.** Use solvability, consistency, leakage and disclosure checks on the sample before deciding on more.

**Be clear about limits.** The material is generated and synthetic. It is not real business data and does not stand in for testing on your own systems. Scope, size, delivery and licence terms are confirmed with MirrorFirm for each engagement, not assumed.

A narrow, well-specified pilot shows whether the material fits your harness and questions before you commit to anything broader.

Audience:
Evaluation teams
Content type:
Explanation
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Reconciliation mismatch