When should asking for clarification count as a successful review response?
Clarification can be the correct outcome when a missing fact controls the decision, but evaluation should distinguish a targeted evidence request from avoidable deferral.
**Should every review task end with a substantive decision?**
No. Some tasks should end with a targeted request for information. If the supplied procedure requires a fact that the available records cannot establish, choosing an outcome anyway can be the central failure.
**What makes a clarification request good?**
It identifies the missing fact, explains why that fact matters and directs the request to a role able to provide it. In a professional services account review, uncertainty about which service remains contracted should produce a request about contract scope, rather than a broad demand for every customer document.
**How do we prevent models from asking unnecessary questions?**
Include cases where the answer already exists in an accessible record. Score whether the agent inspected that source before deferring. Also check whether the requested information could actually change the decision under the supplied rule.
A MirrorFirm task pack could distinguish resolvable uncertainty, genuinely missing evidence and irrelevant gaps. Evaluation can then reward useful clarification while penalising both unsupported certainty and unnecessary requests that create work for another team.
Reward justified uncertainty without encouraging models to avoid decisions they can already make.
- Audience:
- Model labs
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should precede training on generated review cases?
- How can a late event reveal whether an agent checks its assumptions before committing a decision?
- Which evaluation hooks make review runs reproducible and auditable?
- How should evaluation distinguish a sound review decision from a persuasive explanation?
- How can we tell whether an agent responds to meaningful evidence rather than superficial cues?
- How should buyers inspect realism and disclosure risk in generated review material?