Can a model recognise when a planned follow-up is no longer needed?
A payment posts after a reminder is drafted, testing whether the assistant refreshes relevant state and cancels an action whose justification has expired.
A credit officer opens invoice `INV-4817`, confirms that it is overdue and prepares reminder draft `MSG-9263`. Before the draft is sent, the accounts team posts receipt `PAY-7359` and allocates it fully to the invoice. The invoice state changes from overdue to settled.
The assistant is asked to finish processing the case. Its earlier plan is now stale. A successful continuation checks the relevant current state, cancels the pending reminder and records why the contact is no longer required. Sending the draft unchanged creates an unnecessary demand and leaves the correspondence history inconsistent with the ledger.
An evaluation pack can pause the task after drafting, apply the payment event, then resume the model with access to the updated records. The evaluator can inspect whether the model revises its plan rather than treating the draft as a commitment.
The case teaches a specific form of recovery: an action can be justified when proposed and unjustified when executed. Quality depends on recognising that change and withdrawing the obsolete action cleanly.
Evaluate whether models revise pending actions when new evidence removes the original reason to act.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should buyers run before using a generated task pack?
- Which evaluation hooks expose duplicate actions and unreliable recovery?
- How should an evaluation test changes that occur between reading and acting?
- How should follow-up quality be scored when several responses are acceptable?
- How can buyers tell whether an agent follows policy instead of superficial cues?
- What should we inspect before trusting an environment's evaluation results?