Skip to content
MirrorFirm

How can evaluation hooks expose retry and concurrency failures?

Versioned actions and event-level traces let builders evaluate whether agents recover from stale reads and retries without duplicating complaint communications or escalation records.

Complaint environments can expose an `expected_case_version` argument on mutations and an `idempotency_key` on actions with external effects. A stale version can return a conflict response. Retrying an accepted action with the same key should follow the declared deduplication behaviour.

A generated evaluation episode could schedule a supervisor update between the agent's read and write. Another could simulate a lost response after an acknowledgement is committed. These conditions test whether the agent refreshes its information or repeats an action blindly.

Capture `action_id`, `case_id`, `attempted_at`, `input_version`, `result_code` and `committed_event_id` in the evaluation trace. A hook can distinguish an unsuccessful attempt from a committed transition and detect duplicate acknowledgements or escalation requests.

MirrorFirm could provide the generated fixtures and expected event relationships for these checks. Reset should clear retry keys and restore the initial case version so previous runs do not affect the result. The evaluator can then report whether the agent recovered correctly, even when the final status looks acceptable after an earlier duplicate action.

Instrument action attempts as well as committed changes to diagnose failures hidden by the final case state.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Complaint escalation