How should an evaluation test changes that occur between reading and acting?
A payment posts after an agent reads an overdue balance but before it submits a reminder, testing whether the agent checks fresh state.
A collections agent reads invoice `INV-5720` and sees an outstanding balance eligible for a reminder. It retrieves the approved wording and prepares a message. Before submission, a payment allocation event clears the invoice and advances its state version.
The agent's initial decision was supported by the information it read. Its prepared message is now stale. The generated environment requires submission against the current invoice version, so the attempted action returns a state-conflict response instead of creating a contact event.
The agent reloads the invoice, recognises that the obligation is settled and discards the reminder. It records that the case changed during preparation rather than treating the conflict as a reason to retry the same action blindly.
MirrorFirm could generate this event sequence as a simulation task with a controlled interleaving point. Evaluation should inspect whether the agent refreshes the relevant state, changes its decision and avoids contact after settlement. The lesson is about decision freshness: a sound earlier judgement does not automatically authorise a later action against changed business facts.
Evaluate the final action boundary so agents cannot rely indefinitely on a previously valid view of the account.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should buyers run before using a generated task pack?
- Which evaluation hooks expose duplicate actions and unreliable recovery?
- How should follow-up quality be scored when several responses are acceptable?
- How can buyers tell whether an agent follows policy instead of superficial cues?
- What should we inspect before trusting an environment's evaluation results?
- How can we tell whether an agent learned the policy or a dataset shortcut?