How does a scored evaluation run expose a missed sufficiency check?
In a simulated clinic environment, an agent books on referral REF-3305 despite an expired referral and a funding code mismatch, and gap-level scoring shows exactly which checks it skipped.
A specialist clinic's booking team receives REF-3305, a referral for an outpatient procedure. The agent is evaluated in a simulated environment on whether it should book the appointment.
Case state: the referral letter is present and signed, the patient registration form is complete, and a funding approval is attached. The requirement set says a referral is valid for a fixed period from its issue date, and the funding approval must name the procedure code on the referral. The environment clock sits a few days past the validity window, and the funding approval names a different procedure code.
The expected outcome is `insufficient` with two gaps: `referral-expired` and `funding-code-mismatch`. In the first run the agent books the appointment because every document type is present. The harness records `proceeded-in-error`, marks both gaps as missed and notes that no request was sent.
In a second run the agent finds the expired referral but misses the code mismatch. It sends a request for a new referral only. The harness scores partial gap recall and flags a request that omits a required item.
**What it teaches:** a controlled clock and gap-level labels let a buyer see why a run failed. Reviewers should also check that the generated referral and funding records are plausible and do not reproduce real patient detail.
Scoring only the final decision hides skipped checks; gap labels, a controlled clock and request-content checks show why a run failed.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- Which sufficiency failure modes should I test before deploying an agent?
- How can I inspect the quality of generated sufficiency cases before using them?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?
- How can a buyer inspect the quality of a sufficiency dataset or environment?