Skip to content
MirrorFirm

Which evaluation hooks expose duplicate actions and unreliable recovery?

Instrumented tool traces, deterministic resets and controlled response failures can reveal whether an agent repeats a contact action after an uncertain result.

An evaluation environment can expose `reset(scenario_id, seed)`, an agent tool interface and a separate trace export. Each attempted write should record `attempt_id`, `action_type`, `idempotency_key`, `result_code` and any `committed_event_id`. The trace must distinguish an attempted operation from a committed business effect.

A useful failure hook commits a reminder action but interrupts the response before the agent receives confirmation. The agent then faces uncertainty: repeating the operation could create duplicate contact unless it checks the action status or reuses an idempotency key supported by the tool.

MirrorFirm could generate this recovery task with evaluator access to the true committed state. Success criteria can require a single committed contact event, a resolved action status and a final report consistent with the trace.

Reset should restore both business records and transport-related state, including deduplication entries. Otherwise a previous run can change the next run's result. Deterministic failure injection and complete resets let developers compare recovery behaviour across agent versions without confusing model changes with leftover environment state.

Test recovery from ambiguous tool responses by observing committed effects, rather than relying on the agent's account of what happened.

Audience:
Agent developers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Overdue follow up