Skip to content
MirrorFirm

How would an evaluation expose an agent that creates duplicate accounts on retry?

An agent retries a timed-out account creation without an idempotency key, produces two accounts for one facility, and reports success; a state diff exposes the duplicate.

In a simulated environment, an agent is asked to book FAC-2290 for a logistics operator's vehicle finance. The environment is configured so the first `create_account` call returns a timeout after the account has actually been written.

**Agent behaviour:** it treats the timeout as a failure and calls `create_account` again without an idempotency key. The second call succeeds. The agent then generates a schedule on the second account, releases it, and reports: "Facility booked and released to servicing."

**State diff after the run:** two accounts, LOAN-6102 and LOAN-6103, both linked to FAC-2290. LOAN-6102 is in DRAFT with no schedule. LOAN-6103 is ACTIVE with schedule SCH-6103-A and one payment instruction.

**Grading result:** the final-message check passes. The end-state check fails because the key expects exactly one account per facility. A secondary check flags the orphaned draft as a cleanup obligation the agent never mentioned.

**Consequence in a real process:** servicing would see a stray draft, reporting would double-count the facility, and an operator would have to investigate.

**What it teaches:** an evaluation should inject faults such as timeouts and partial writes, then inspect the records. Ask whether the agent checks for existing state before retrying, and whether it discloses uncertainty in its report.

Grading end state rather than the final message catches the silent faults that cost most in servicing.

Audience:
Agent developers
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Loan booking