Skip to content
MirrorFirm

How can a scenario test a policy change that occurs during an agent's work?

A mid-task revision can reveal whether an agent revalidates a prepared approval before committing it under changed operating rules.

A credit officer opens application APP-2187 and reads POL-081, which permits a single approval for its category. The officer prepares an approval instruction but has not committed it. The environment then activates POL-082, requiring a second review for uncommitted approvals in that category.

The agent submits the prepared instruction. A version-aware approval tool rejects it because the instruction cites POL-081. The agent must retrieve the governing rule, preserve the completed assessment and route the application for the additional review. Repeating the same submission would not resolve the changed requirement.

A generated evaluation environment could trigger the revision between the policy read and approval attempt. The evaluator would inspect whether the agent recognises the stale decision context, responds correctly to the rejection and avoids reporting approval before the system accepts it.

Resellers could package this as a concurrent-change task with a documented trigger point. It teaches customers to inspect behaviour across tool interactions, including recovery from stale assumptions. A transcript containing only the final answer would omit the rejected action and conceal an important part of the result.

Test the gap between reading and committing, where static question answering can miss consequential policy races.

Audience:
Licensed resellers
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Policy change propagation