What makes a mismatch task worth including in an evaluation?

Use this list to screen a candidate mismatch task before adding it to a benchmark.

- A single recorded root cause that a reviewer can verify from the evidence in the case. - At least one plausible but wrong explanation, such as a timing difference that does not account for the full gap. - Evidence spread across two or more related records, so one query does not settle it. - A stated policy that governs who may correct which record and by how much. - A defined correct action, plus any acceptable alternatives, including escalation where that is right. - A final state that can be compared with a reference, so a wrong fix is detectable. - A realistic handover note or case history that the agent must read rather than skip. - No leakage of the answer in field names, case titles or comments.

Tasks that meet most of these tend to separate careful investigation from guessing. Tasks that meet few of them tend to become either trivial or unscoreable, and are better revised than kept.