What does scoping a pilot look like for a team evaluating logistics agents?

An evaluation lead at a freight forwarder is choosing whether to build a benchmark for invoice-checking agents. Their finance operations colleagues spend time resolving differences between carrier invoices and shipment records, so they ask MirrorFirm for a pilot sample.

They scope it to one pair of systems: a transport management system and carrier invoices. The request lists four cause types: a fuel surcharge applied after the quote, a duplicate invoice `FINV-4410` against `FINV-4381`, a weight unit mismatch, and a shipment split across two invoices.

They ask for the policy that sets a tolerance for surcharge variance, an approval limit for credit requests, and cases where the right action is to escalate to the carrier relationship owner. They request ground truth with required evidence ids and ask for the cases as JSONL.

Before deciding, they set a test: reviewers solve a subset without the answer key, and a baseline agent is run on all cases. If reviewers solve most and the baseline fails on a clear subset, they proceed to a stateful environment.

**What it teaches**

- The pilot is tied to a decision and a pass condition. - The scope names cause types, policies and labels, not just volume. - Inspection by reviewers comes before any agent scoring.