How does separate scoring change the comparison of two agents on a referral file?
Two agents reach the same final status on a specialist referral, but gap-level scoring shows one misses a stale medication list and the other over-requests.
An evaluation lead at a health service is comparing two agents on a set of referral cases. One case is REF-6120, a request for a specialist outpatient appointment. The referral letter and recent pathology results are on file. The clinic's requirement set also calls for a signed patient consent form and, for this referral type, a medication list dated within 90 days.
Agent A reads the file, sees the consent form is missing and sends a request for consent only. The medication list on file is nine months old, and Agent A did not check its date. Agent B requests consent and a current medication list, and also re-requests the pathology results that were already supplied.
Scoring on final status alone rates both as AWAITING_INFO and calls them equal. The lead instead scores against the gold record, where the unmet requirements are consent and medication currency. Agent A names one of two and misses the stale list. Agent B names both but adds an unnecessary item.
Across the set, the lead sees that Agent A is fluent but weak on validity rules, while Agent B over-requests. The first needs training cases with expiry defects. The second needs cases where earlier evidence is clearly satisfied.
The consequence: a deployment decision based on final status would have hidden both patterns. **What it teaches:** score identified gaps, unnecessary requests and validity checks as separate measures.
Final status alone can rate different behaviours as equal; scoring gaps, extra requests and validity checks separately shows what each agent needs to improve.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- How does a scored evaluation run expose a missed sufficiency check?
- Which sufficiency failure modes should I test before deploying an agent?
- How can I inspect the quality of generated sufficiency cases before using them?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?