What does a first sample review look like for a healthcare remittance task set?

An evaluation team lead at a benchmark group wants to test agents on remittance matching in a healthcare billing setting. They send a scoping note asking for a small sample of tasks with these properties: insurer payments against patient accounts, remittance advice that covers several claims, and short payments with adjustment codes.

The sample arrives as a task file, snapshot tables and a policy document. The lead's engineer loads the snapshot into the team's harness, resets it, and runs a baseline agent on the first group of tasks. A reviewer works five tasks by hand without the reference.

Two findings follow. The reviewer disagrees with the reference on TASK-0207, where policy does not say whether an adjustment code should reduce the open amount before or after allocation. Separately, the engineer finds that claim identifiers in the remittance text use two formats, which breaks a simple lookup.

The lead sends both findings back with the task ids. The policy text is clarified and the identifier formats are documented as deliberate noise. The state change is a revised sample, VER-2, issued with a change note.

The consequence of skipping this step would have been a full-scale run in which the ambiguous task silently penalised correct agents.

**What it teaches:** a first step should be small enough to review by hand, should be tested in your own harness, and should include a route for sending defects back. Ask for versioned samples and change notes.