How can evaluation hooks expose retry and concurrency failures?
Versioned actions and event-level traces let builders evaluate whether agents recover from stale reads and retries without duplicating complaint communications or escalation records.
Complaint environments can expose an `expected_case_version` argument on mutations and an `idempotency_key` on actions with external effects. A stale version can return a conflict response. Retrying an accepted action with the same key should follow the declared deduplication behaviour.
A generated evaluation episode could schedule a supervisor update between the agent's read and write. Another could simulate a lost response after an acknowledgement is committed. These conditions test whether the agent refreshes its information or repeats an action blindly.
Capture `action_id`, `case_id`, `attempted_at`, `input_version`, `result_code` and `committed_event_id` in the evaluation trace. A hook can distinguish an unsuccessful attempt from a committed transition and detect duplicate acknowledgements or escalation requests.
MirrorFirm could provide the generated fixtures and expected event relationships for these checks. Reset should clear retry keys and restore the initial case version so previous runs do not affect the result. The evaluator can then report whether the agent recovered correctly, even when the final status looks acceptable after an earlier duplicate action.
Instrument action attempts as well as committed changes to diagnose failures hidden by the final case state.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should developers run before training on generated complaint material?
- How can a task test escalation when an earlier case note is misleading?
- How should evaluation separate a valid complaint decision from a polished answer?
- Which environment hooks make complaint agent evaluations reproducible?
- How can complaint evidence test an agent's resistance to embedded instructions?
- How can complaint scenarios test resistance to instructions inside evidence?