How might a reseller's quality check catch answers leaking into a sample's inputs?
A marketplace quality lead notices a baseline scoring too well, traces it to a note field that restates the answer, and has labels separated before the sample is approved.
A marketplace's data quality lead receives a sample for a retail wholesaler's payment matching. It contains `bank_lines.jsonl`, `invoices.parquet` and `cases.jsonl`. They run a baseline that predicts the invoice from the case inputs.
The baseline scores almost perfectly, which is the first warning. The lead inspects `BL-7740`, a bank line with `raw_reference` set to 'INV-61002 part'. Case `CASE-0412` asks which invoice the payment clears, and the same answer appears in a `debug_note` field inside the case input: 'resolved to INV-61002'. The answer is sitting in the question.
The lead logs the defect against the sample. The labels are moved into a separate `labels.jsonl`, the note is removed from the case inputs and the sample is reissued. On the rerun, the baseline falls on cases where the reference is missing or names two invoices, and a stronger system separates itself from it.
**State change:** the sample's status in the marketplace moves from held to approved for preview, with a note recording the check that was added.
**What it teaches:** run a trivial baseline before listing. If it does well, look for leaked labels, fields that restate the answer, or references too clean to be representative. Add a leak check to your acceptance routine for every release.
Run a trivial baseline before listing; a surprisingly high score is a prompt to look for leaked labels and overly clean references.
- Audience:
- Licensed resellers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do I judge whether generated remittance and allocation records are realistic enough?
- Which failure modes should I test for in a payment matching agent?
- How would an evaluation score an agent on a hospital insurer remittance?
- What reset, logging and grading hooks should an evaluation environment expose?
- How should scoring handle cases where several allocations are defensible?
- How do I check generated data for grounding and disclosure risk?