Skip to content
MirrorFirm

How can a scenario test whether a model mistakes repeated evidence for independent support?

An insurance review bundle contains several copies of one assessment, testing whether the model recognises shared origin instead of counting attachments as separate corroboration.

An insurance review officer gathers evidence that a repair assessment received an independent technical check. CASE-0618 contains an assessor report, an email attachment and a document exported from the claims portal. Each repeats the same assessment text.

A document processing action gives the portal export a new file identifier. The bundle state changes from containing one candidate record to containing several candidates, and a model may describe the assessment as corroborated by multiple sources. The lineage fields show that every candidate derives from REP-2187.

The consequence is a false impression of independent review. None of the copies records a separate technical check. A defensible answer groups the copies by origin, chooses a suitable representative and identifies the missing independent review evidence.

This generated scenario supports evaluation of a specific failure mode: confusing file multiplicity with evidential independence. A scoring rule can assess whether the model recognises the common origin and avoids claiming corroboration. It should still allow duplicate copies to serve a legitimate purpose, such as showing where a record was distributed, when that is the claim under review.

Test source independence so repeated copies cannot create a false impression of stronger evidence.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Audit evidence assembly