How does a graded run expose a skipped completeness check?
An evaluation engineer runs an agent on a clinic referral task, and structured expected outcomes show it routed an unsigned referral, pinpointing a document-checking weakness.
An evaluation engineer runs a candidate agent against TASK-0931, a referral intake task for a specialist clinic's administration team. Referral REF-0931 arrives by fax-to-email with a patient name, a referring practitioner and a clinical summary. The referral policy requires a signed referral dated within a stated validity window and a provider number.
The attached scan has a provider number but no signature. The agent sets REF-0931 to READY_FOR_TRIAGE and assigns it to the triage queue, and its note says "referral complete".
The grader compares the exported state with the expected block: `final_status` should be PENDING_INFO, `reason_codes` should include MISSING_SIGNATURE, and `message_type` should be `request_to_referrer`. Three fields differ. The grader also flags a rule violation, since the forbidden action list includes `assign_queue` before completeness passes.
The consequence in the run report is a failed task with the failure labelled by type: a completeness check skipped, not a routing error. The engineer then looks across the other tasks with signature or date defects and finds the same pattern, which points to the agent's document-checking step rather than to its language handling.
**What it teaches:** structured expected outcomes let a failure be located. A pass or fail score alone says little; the failure class tells the agent developer where to work next.
Structured expected outcomes locate the failure class, so developers learn where to work rather than only seeing a failed score.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What should you check when you receive an intake sample?
- What evaluation hooks should a simulated intake environment expose?
- What failure modes should an intake evaluation measure?
- How can a buyer judge the quality of generated intake data?
- How can a buyer inspect the quality of a generated intake environment?
- How can a run that reaches the right queue still be a failure?