Skip to content
MirrorFirm

How does separate scoring change the comparison of two agents on a referral file?

Two agents reach the same final status on a specialist referral, but gap-level scoring shows one misses a stale medication list and the other over-requests.

An evaluation lead at a health service is comparing two agents on a set of referral cases. One case is REF-6120, a request for a specialist outpatient appointment. The referral letter and recent pathology results are on file. The clinic's requirement set also calls for a signed patient consent form and, for this referral type, a medication list dated within 90 days.

Agent A reads the file, sees the consent form is missing and sends a request for consent only. The medication list on file is nine months old, and Agent A did not check its date. Agent B requests consent and a current medication list, and also re-requests the pathology results that were already supplied.

Scoring on final status alone rates both as AWAITING_INFO and calls them equal. The lead instead scores against the gold record, where the unmet requirements are consent and medication currency. Agent A names one of two and misses the stale list. Agent B names both but adds an unnecessary item.

Across the set, the lead sees that Agent A is fluent but weak on validity rules, while Agent B over-requests. The first needs training cases with expiry defects. The second needs cases where earlier evidence is clearly satisfied.

The consequence: a deployment decision based on final status would have hidden both patterns. **What it teaches:** score identified gaps, unnecessary requests and validity checks as separate measures.

Final status alone can rate different behaviours as equal; scoring gaps, extra requests and validity checks separately shows what each agent needs to improve.

Audience:
Evaluation teams
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Document sufficiency