Skip to content
MirrorFirm

Which evaluation hooks expose retry and duplicate-processing failures?

Deterministic resets, injected tool failures and observable side effects let developers test whether retries complete a change without producing duplicate business actions.

A simulated environment can expose `reset(initial_state_id, seed)` and retain an append-only action log for each run. Reset semantics should restore records, pending messages, simulated time and deduplication state; restoring only database rows can leave later runs contaminated by earlier activity.

A failure hook can make `submit_change` commit successfully while its response times out. The agent then faces an uncertain outcome. A sound recovery uses an operation identifier or a status lookup before deciding whether another submission is needed.

Evaluation hooks can count business side effects by `operation_id`, inspect `idempotency_key` reuse and distinguish repeated tool calls from duplicated actions. Calling a tool twice may be harmless; creating two replacement delivery bookings is not.

Buyers can request failure schedules that are reproducible and hidden from the agent. The harness can record the injection point alongside the action trace for diagnosis. This makes it possible to assess whether recovery preserves exactly the intended business outcome even when transport behaviour gives the agent incomplete information about what happened.

Measure recovery at the side-effect boundary, where an uncertain tool response can conceal a successful write.

Audience:
Agent developers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Customer change handling