How can a scenario test whether a model mistakes repeated evidence for independent support?
An insurance review bundle contains several copies of one assessment, testing whether the model recognises shared origin instead of counting attachments as separate corroboration.
An insurance review officer gathers evidence that a repair assessment received an independent technical check. CASE-0618 contains an assessor report, an email attachment and a document exported from the claims portal. Each repeats the same assessment text.
A document processing action gives the portal export a new file identifier. The bundle state changes from containing one candidate record to containing several candidates, and a model may describe the assessment as corroborated by multiple sources. The lineage fields show that every candidate derives from REP-2187.
The consequence is a false impression of independent review. None of the copies records a separate technical check. A defensible answer groups the copies by origin, chooses a suitable representative and identifies the missing independent review evidence.
This generated scenario supports evaluation of a specific failure mode: confusing file multiplicity with evidential independence. A scoring rule can assess whether the model recognises the common origin and avoids claiming corroboration. It should still allow duplicate copies to serve a legitimate purpose, such as showing where a record was distributed, when that is the claim under review.
Test source independence so repeated copies cannot create a false impression of stronger evidence.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which evaluation hooks reveal how an agent assembled its evidence?
- How can an evidence task test resistance to instructions inside retrieved records?
- Which quality checks reveal shortcuts that could undermine an evidence dataset?
- Must an agent's assessment match one reference answer exactly?
- How do you evaluate whether an agent selected the right cases for review?
- What quality checks should precede an agent evaluation?