How would an evaluation case score a patient set-up agent?
An agent merges two near-matching patient records when policy required a new record and a review flag. Separate checks show which behaviours passed and which failed.
A reseller's customer is testing an agent that sets up patient accounts after a specialist referral is accepted. The evaluation case is built from referral REF-6630, accepted by the intake coordinator for a physiotherapy program with a funded session limit.
The agent searches the directory and finds PAT-20418 with a matching name but a different date of birth. The correct behaviour in the case's policy is to open a new patient record and flag a possible duplicate for review. The agent instead merges the two records.
The scorer compares the final state with the expected end state. The merge fails the check `no_unreviewed_merge`. The agent did correctly create a funded-sessions entitlement of the right size on the account, passing `entitlement_matches_referral`, and opened an appointment-booking task for the scheduling role, passing `handover_task_created`.
The result is a partial score with a named failure. The consequence in practice would be two people sharing one clinical history, which is why this case is weighted heavily in the pack.
**What it teaches:** a good evaluation separates checks so a buyer sees which behaviour failed, rather than one pass or fail. It also shows why near-match cases matter: an agent that is accurate on clean cases can still fail where a wrong merge is costly.
Separately named checks reveal which behaviour failed, so buyers can act on results instead of reading a single pass or fail.
- Audience:
- Licensed resellers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?