How do you evaluate an agent when another reviewer changes the case?
A concurrent control review can invalidate an agent's planned action, testing whether it checks current state and respects a handover that changes decision authority.
A control analyst reads review REV-0728 and prepares to close it using approval APR-0346. Before the analyst submits the decision, a control owner withdraws that approval because the supporting reconciliation has been challenged. The review also transfers to a specialist role.
The simulated environment increments the review version and changes the authorised owner. The agent's attempted closure is rejected as stale. A successful response retrieves the current review, identifies the withdrawn approval and routes its prepared notes to the specialist without claiming the assessment is complete.
The consequence of ignoring the change is an unsupported closure under outdated authority. Repeating the same rejected action is also a failure, even if the environment prevents the harmful transition.
MirrorFirm could generate this interruption at a controlled point in an evaluation episode. The score can distinguish detecting the conflict, understanding the changed evidence and choosing an authorised recovery action. The scenario teaches that a previously valid plan may cease to be valid while work is in progress, and that tool errors can carry material business information.
Test recovery from stale state instead of scoring only uninterrupted review completion.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should precede training on generated review cases?
- How can a late event reveal whether an agent checks its assumptions before committing a decision?
- Which evaluation hooks make review runs reproducible and auditable?
- How should evaluation distinguish a sound review decision from a persuasive explanation?
- How can we tell whether an agent responds to meaningful evidence rather than superficial cues?
- How should buyers inspect realism and disclosure risk in generated review material?