How can tool hooks detect duplicate escalation after an uncertain response?
An evaluation can simulate a lost tool response after a successful write, testing whether the agent checks state before creating a duplicate escalation.
A simulated environment could accept `submit_escalation` and persist the escalation, then return a timeout instead of confirmation. This creates a specific uncertainty: the agent knows the response failed but does not yet know whether the business action failed.
The tool contract can support an `idempotency_key`, while a read action exposes escalations linked to the case. A capable recovery path checks the existing action or retries with the same key. Submitting a fresh escalation blindly can create duplicate work for the receiving team.
Evaluation hooks can capture `request_id`, `idempotency_key`, `case_id`, `committed_action_id` and `response_delivery_status`. The scorer should distinguish repeated transport attempts from multiple committed business actions. It can also check whether the agent's final message accurately describes the confirmed state.
For reproducibility, reset must restore both the case and the action-key store before each run. A fault schedule should specify where the response is lost. MirrorFirm could scope generated tasks around this mechanism to test operational recovery without confusing a tool transport error with an authorised reason to abandon the complaint.
Test recovery from ambiguous tool outcomes using durable action records and duplicate detection.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should developers run before training on generated complaint material?
- How can a task test escalation when an earlier case note is misleading?
- How should evaluation separate a valid complaint decision from a polished answer?
- Which environment hooks make complaint agent evaluations reproducible?
- How can complaint evidence test an agent's resistance to embedded instructions?
- How can evaluation hooks expose retry and concurrency failures?