Skip to content
MirrorFirm

How do we check that a complaint benchmark is not revealing its answers?

Benchmark inspection should look for answer leakage in metadata, document ordering and hidden-state exposure, not just explicit labels inside the complaint text.

**Where can answer leakage appear?**

A file name may identify the intended escalation route. A case identifier may encode the outcome. Documents may always place decisive evidence last. Internal metadata may include a final disposition that the acting role could not yet know.

**What should the buyer inspect?**

Ask for a field-level distinction between model-visible inputs, environment-only state and evaluator-only labels. Review exported prompts and tool responses, because a clean source dataset can still leak answers when an integration assembles the task.

**How can we test for shortcuts?**

Run a restricted-input baseline that receives identifiers and structural metadata without complaint evidence. If it predicts outcomes, investigate which artefacts carry the signal. Shuffle non-semantic document order where order is not itself part of the task.

**What about train and evaluation splits?**

Related variants should be grouped before splitting. Otherwise, a model may encounter the same underlying case with lightly changed wording during training. MirrorFirm could supply generated case-family identifiers for split management while excluding those identifiers from the model's observation. This supports an inspection of memorisation risk without turning family membership into another answer cue.

Keep evaluation difficulty tied to investigation and decision-making by separating observable evidence from scoring information.

Audience:
Model labs
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Complaint escalation