How do we check that a complaint benchmark is not revealing its answers?
Benchmark inspection should look for answer leakage in metadata, document ordering and hidden-state exposure, not just explicit labels inside the complaint text.
**Where can answer leakage appear?**
A file name may identify the intended escalation route. A case identifier may encode the outcome. Documents may always place decisive evidence last. Internal metadata may include a final disposition that the acting role could not yet know.
**What should the buyer inspect?**
Ask for a field-level distinction between model-visible inputs, environment-only state and evaluator-only labels. Review exported prompts and tool responses, because a clean source dataset can still leak answers when an integration assembles the task.
**How can we test for shortcuts?**
Run a restricted-input baseline that receives identifiers and structural metadata without complaint evidence. If it predicts outcomes, investigate which artefacts carry the signal. Shuffle non-semantic document order where order is not itself part of the task.
**What about train and evaluation splits?**
Related variants should be grouped before splitting. Otherwise, a model may encounter the same underlying case with lightly changed wording during training. MirrorFirm could supply generated case-family identifiers for split management while excluding those identifiers from the model's observation. This supports an inspection of memorisation risk without turning family membership into another answer cue.
Keep evaluation difficulty tied to investigation and decision-making by separating observable evidence from scoring information.
- Audience:
- Model labs
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should developers run before training on generated complaint material?
- How can a task test escalation when an earlier case note is misleading?
- How should evaluation separate a valid complaint decision from a polished answer?
- Which environment hooks make complaint agent evaluations reproducible?
- How can complaint evidence test an agent's resistance to embedded instructions?
- How can evaluation hooks expose retry and concurrency failures?