How would an evaluation expose an agent that creates duplicate accounts on retry?
An agent retries a timed-out account creation without an idempotency key, produces two accounts for one facility, and reports success; a state diff exposes the duplicate.
In a simulated environment, an agent is asked to book FAC-2290 for a logistics operator's vehicle finance. The environment is configured so the first `create_account` call returns a timeout after the account has actually been written.
**Agent behaviour:** it treats the timeout as a failure and calls `create_account` again without an idempotency key. The second call succeeds. The agent then generates a schedule on the second account, releases it, and reports: "Facility booked and released to servicing."
**State diff after the run:** two accounts, LOAN-6102 and LOAN-6103, both linked to FAC-2290. LOAN-6102 is in DRAFT with no schedule. LOAN-6103 is ACTIVE with schedule SCH-6103-A and one payment instruction.
**Grading result:** the final-message check passes. The end-state check fails because the key expects exactly one account per facility. A secondary check flags the orphaned draft as a cleanup obligation the agent never mentioned.
**Consequence in a real process:** servicing would see a stray draft, reporting would double-count the facility, and an operator would have to investigate.
**What it teaches:** an evaluation should inject faults such as timeouts and partial writes, then inspect the records. Ask whether the agent checks for existing state before retrying, and whether it discloses uncertainty in its report.
Grading end state rather than the final message catches the silent faults that cost most in servicing.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How do I know whether my success criteria for booking tasks are strict enough?
- Which failure modes should I probe when evaluating agents on facility booking?