Must an agent's assessment match one reference answer exactly?
Evaluation can accept different defensible assessments while requiring supported claims, consistent reasoning and explicit recognition of material uncertainty.
**Should every successful answer use the same wording and conclusion?**
No. Some evidence sets support more than one defensible recommendation, particularly when a supplied policy leaves room for reviewer judgement. Exact narrative matching can reward imitation while penalising a well-supported alternative.
**What should remain fixed?**
The evaluator can require accurate factual claims, recognition of material contradictions and a clear separation between established facts and judgement. It can also specify conclusions that the evidence cannot support, regardless of wording.
**How can a buyer inspect the scoring approach?**
Request an assessment rubric with acceptable reasoning paths, disallowed claims and examples of uncertainty that must be surfaced. Ask how a concise qualified answer compares with a confident answer that overlooks a limitation.
For contested cases, retain a route for human adjudication and record the reason for any scoring decision. A generated task pack could include alternative reference rationales rather than a single model paragraph. This helps developers evaluate evidence-based judgement without treating stylistic similarity as a measure of correctness.
Use claim-level assessment criteria so valid alternative reasoning is not penalised for differing from a reference narrative.
- Audience:
- Agent developers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which evaluation hooks reveal how an agent assembled its evidence?
- How can an evidence task test resistance to instructions inside retrieved records?
- Which quality checks reveal shortcuts that could undermine an evidence dataset?
- How do you evaluate whether an agent selected the right cases for review?
- What quality checks should precede an agent evaluation?
- How should an evaluation handle an agent that discovers its own earlier mistake?