How should a task be scored when the evidence allows more than one conclusion?
Ambiguous cases need bounded acceptable answers and explicit unresolved facts, allowing customers to evaluate justified uncertainty without requiring a single preferred sentence.
**How should a task be scored when the evidence allows more than one conclusion?**
Use a rubric that separates established facts, unresolved facts and permissible decisions. A single reference paragraph can hide legitimate alternatives and reward wording rather than judgement.
**What would the rubric contain?**
For professional services engagement ENG-2648, the evidence may establish that a deliverable was reviewed but leave the reviewer's delegated authority unclear. Acceptable responses could request the delegation record or mark authorisation as unresolved. A response stating that the deliverable lacked review would contradict the supplied evidence.
The rubric could list required observations, prohibited assertions and acceptable next actions. It could also specify which additional fact would resolve the uncertainty. This makes an incomplete answer distinguishable from an appropriately qualified one.
**How does a reseller make this usable?**
Ask for adjudication notes explaining why each accepted outcome is defensible under the task rules. Customers can use those notes to audit automated graders and resolve scoring disputes. MirrorFirm could scope ambiguous cases separately from determinate cases so a customer's aggregate result does not conceal which kind of judgement was tested.
Offer scoring guidance that distinguishes defensible uncertainty from evasive or unsupported answers.
- Audience:
- Licensed resellers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which evaluation hooks reveal how an agent assembled its evidence?
- How can an evidence task test resistance to instructions inside retrieved records?
- Which quality checks reveal shortcuts that could undermine an evidence dataset?
- Must an agent's assessment match one reference answer exactly?
- How do you evaluate whether an agent selected the right cases for review?
- What quality checks should precede an agent evaluation?