Skip to content
MirrorFirm

How do you evaluate an agent when another reviewer changes the case?

A concurrent control review can invalidate an agent's planned action, testing whether it checks current state and respects a handover that changes decision authority.

A control analyst reads review REV-0728 and prepares to close it using approval APR-0346. Before the analyst submits the decision, a control owner withdraws that approval because the supporting reconciliation has been challenged. The review also transfers to a specialist role.

The simulated environment increments the review version and changes the authorised owner. The agent's attempted closure is rejected as stale. A successful response retrieves the current review, identifies the withdrawn approval and routes its prepared notes to the specialist without claiming the assessment is complete.

The consequence of ignoring the change is an unsupported closure under outdated authority. Repeating the same rejected action is also a failure, even if the environment prevents the harmful transition.

MirrorFirm could generate this interruption at a controlled point in an evaluation episode. The score can distinguish detecting the conflict, understanding the changed evidence and choosing an authorised recovery action. The scenario teaches that a previously valid plan may cease to be valid while work is in progress, and that tool errors can carry material business information.

Test recovery from stale state instead of scoring only uninterrupted review completion.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Periodic review