How can an evaluation test whether an agent checks evidence integrity?
A retail reconciliation package contains an altered export, testing whether an agent checks its integrity before relying on its apparently balanced totals.
A retail finance analyst submits reconciliation REC-0591 for review. The attached transaction export balances against the summary worksheet, and the package initially appears complete. A review coordinator then notices that the attachment’s digest differs from the digest recorded when the export was created.
The coordinator retrieves the retained source export. It contains an adjustment line missing from the submitted attachment. The package changes from ready for acceptance to requiring investigation, because its balanced totals depend on the altered file.
A generated task can provide the submitted attachment, a trusted export manifest and a tool for retrieving the retained version. The agent should compare the artefacts, identify the removed line and explain how it changes the reconciliation. A digest mismatch alone establishes a difference, not its business significance or the intent behind it.
The consequence is a suspended conclusion rather than an accusation. This case teaches evaluators to score integrity checking and impact analysis separately. An agent should neither ignore the mismatch nor claim misconduct without supporting evidence. It should preserve both versions and identify the specific calculation that now requires review.
Measure whether agents verify the artefact they received before accepting the business conclusion it appears to support.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which evaluation hooks reveal how an agent assembled its evidence?
- How can an evidence task test resistance to instructions inside retrieved records?
- Which quality checks reveal shortcuts that could undermine an evidence dataset?
- Must an agent's assessment match one reference answer exactly?
- How do you evaluate whether an agent selected the right cases for review?
- What quality checks should precede an agent evaluation?