How should an evaluation handle an agent that discovers its own earlier mistake?
A utility review tests whether an agent corrects an overstated evidence claim, updates dependent conclusions and explains the revision before resubmission.
A utility review agent links completion certificate CER-2187 to service cases CASE-0412 and CASE-0413. It marks both cases as supported and submits package PKG-0631. The reviewer returns the package because the certificate covers only the first service address.
The environment moves the package from submitted to returned and exposes the review comment. The agent can now inspect the certificate's address field, remove the unsupported link and reassess the second case. Its correction also needs to update the package statement that all listed service completions have supporting certificates.
The second case becomes unresolved, while the first remains supported. The revised submission explains that the earlier coverage statement was too broad and identifies the remaining evidence request. The consequence is a narrower but defensible package instead of a superficially complete one.
This scenario tests recovery as an observable sequence. Evaluation can inspect whether the agent corrected the link, propagated the change to the package conclusion and retained the review history. Merely adding an apology should not count as correction. Equally, discarding all valid evidence would be an unnecessary loss of progress. The useful behaviour is a targeted repair with clear consequences.
Evaluate recovery from a mistaken assessment, including the correction of conclusions that depended on it.
- Audience:
- Environment builders
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which evaluation hooks reveal how an agent assembled its evidence?
- How can an evidence task test resistance to instructions inside retrieved records?
- Which quality checks reveal shortcuts that could undermine an evidence dataset?
- Must an agent's assessment match one reference answer exactly?
- How do you evaluate whether an agent selected the right cases for review?
- What quality checks should precede an agent evaluation?