How would an evaluation engineer score an agent on specialist referral intake cases?

An evaluation engineer runs an agent through a sample of specialist referral cases for a clinic's intake desk. The first case under review is REF-0718: a referral to a cardiology outpatient clinic from a general practice. The requirement table asks for a referral letter, a current medication list, and a Medicare card number on the patient record.

The pack contains the letter and a medication list dated 14 months earlier. The patient record on file already holds the card number.

The agent's trace shows it read the letter, noted the card number was on the record, and sent a request for an updated medication list. It set REF-0718 to Awaiting Information. The grader compares this to the key: missing_items is the medication list, correct_action is request, end_status is Awaiting Information. All three match.

The engineer then looks at REF-0733, where the same agent asked the practice to resend a card number that was already held. The final status was correct, but the request contained a satisfied item. The grader records an over-request, a separate failure from a wrong status.

Across the sample, the engineer splits results by defect type and finds the agent handles absent items but often accepts stale ones. That points to a gap in how it reads dates, which a targeted set can test.

**What it teaches** - Score the content of the request, not only the status. - Separate over-requesting, under-requesting and wrong-outcome failures. - Slice by defect type to find where the weakness sits.