Skip to content
MirrorFirm

How does a graded run expose a skipped completeness check?

An evaluation engineer runs an agent on a clinic referral task, and structured expected outcomes show it routed an unsigned referral, pinpointing a document-checking weakness.

An evaluation engineer runs a candidate agent against TASK-0931, a referral intake task for a specialist clinic's administration team. Referral REF-0931 arrives by fax-to-email with a patient name, a referring practitioner and a clinical summary. The referral policy requires a signed referral dated within a stated validity window and a provider number.

The attached scan has a provider number but no signature. The agent sets REF-0931 to READY_FOR_TRIAGE and assigns it to the triage queue, and its note says "referral complete".

The grader compares the exported state with the expected block: `final_status` should be PENDING_INFO, `reason_codes` should include MISSING_SIGNATURE, and `message_type` should be `request_to_referrer`. Three fields differ. The grader also flags a rule violation, since the forbidden action list includes `assign_queue` before completeness passes.

The consequence in the run report is a failed task with the failure labelled by type: a completeness check skipped, not a routing error. The engineer then looks across the other tasks with signature or date defects and finds the same pattern, which points to the agent's document-checking step rather than to its language handling.

**What it teaches:** structured expected outcomes let a failure be located. A pass or fail score alone says little; the failure class tells the agent developer where to work next.

Structured expected outcomes locate the failure class, so developers learn where to work rather than only seeing a failed score.

Audience:
Agent developers
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Application intake