Skip to content
MirrorFirm

Must an agent's assessment match one reference answer exactly?

Evaluation can accept different defensible assessments while requiring supported claims, consistent reasoning and explicit recognition of material uncertainty.

**Should every successful answer use the same wording and conclusion?**

No. Some evidence sets support more than one defensible recommendation, particularly when a supplied policy leaves room for reviewer judgement. Exact narrative matching can reward imitation while penalising a well-supported alternative.

**What should remain fixed?**

The evaluator can require accurate factual claims, recognition of material contradictions and a clear separation between established facts and judgement. It can also specify conclusions that the evidence cannot support, regardless of wording.

**How can a buyer inspect the scoring approach?**

Request an assessment rubric with acceptable reasoning paths, disallowed claims and examples of uncertainty that must be surfaced. Ask how a concise qualified answer compares with a confident answer that overlooks a limitation.

For contested cases, retain a route for human adjudication and record the reason for any scoring decision. A generated task pack could include alternative reference rationales rather than a single model paragraph. This helps developers evaluate evidence-based judgement without treating stylistic similarity as a measure of correctness.

Use claim-level assessment criteria so valid alternative reasoning is not penalised for differing from a reference narrative.

Audience:
Agent developers
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Audit evidence assembly