Skip to content
MirrorFirm

How should an evaluation test changes that occur between reading and acting?

A payment posts after an agent reads an overdue balance but before it submits a reminder, testing whether the agent checks fresh state.

A collections agent reads invoice `INV-5720` and sees an outstanding balance eligible for a reminder. It retrieves the approved wording and prepares a message. Before submission, a payment allocation event clears the invoice and advances its state version.

The agent's initial decision was supported by the information it read. Its prepared message is now stale. The generated environment requires submission against the current invoice version, so the attempted action returns a state-conflict response instead of creating a contact event.

The agent reloads the invoice, recognises that the obligation is settled and discards the reminder. It records that the case changed during preparation rather than treating the conflict as a reason to retry the same action blindly.

MirrorFirm could generate this event sequence as a simulation task with a controlled interleaving point. Evaluation should inspect whether the agent refreshes the relevant state, changes its decision and avoids contact after settlement. The lesson is about decision freshness: a sound earlier judgement does not automatically authorise a later action against changed business facts.

Evaluate the final action boundary so agents cannot rely indefinitely on a previously valid view of the account.

Audience:
Agent developers
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Overdue follow up