Which evidence imperfections should a generated review pack contain?
Deliberate evidence imperfections can test whether an agent handles incomplete, stale and contradictory material without confusing accidental dataset defects with intended task difficulty.
Ask for evidence imperfections that each exercise a defined review behaviour and have a clear explanation in evaluator material.
- Include an expired certificate whose replacement exists elsewhere, requiring retrieval before the requirement can be treated as satisfied. - Include a submitted document with a missing attachment, leaving a specific requirement unsupported despite a reassuring cover message. - Include conflicting declarations from different reporting periods, requiring the agent to distinguish a change from an actual contradiction. - Include a duplicate upload with a different filename, testing whether repeated evidence is mistaken for independent corroboration. - Include a corrected record that clearly supersedes an earlier version, requiring the review to use the effective replacement. - Include an unreadable evidence item with a defined recovery route, testing whether the agent requests usable material.
The generated pack should identify which imperfections are intentional and which conclusions they affect. Otherwise, evaluation teams may spend effort diagnosing broken fixtures rather than agent behaviour. Buyers can also request clean counterparts so that difficulty can be attributed to a particular evidence problem instead of several uncontrolled changes.
Specify evidence defects as controlled task features with documented consequences for review decisions.
- Audience:
- Evaluation teams
- Content type:
- Checklist
- Buyer question:
- What can be generated
Content release: r-a220eb6e099b243b
Related questions
- How could a simulated environment represent review actions and their permitted transitions?
- What generated state could connect a carrier review to corrective operational work?
- What evidence relationships could a generated review dataset contain?
- Can a task pack include policies that change while a review is open?
- What should a generated workload contain to test review queue management?
- How can generated relationships make a customer review more demanding?