How would a grader catch a model that skips an approval condition?
A health administration provider set-up task shows how a state-based grader flags a model that registers a practitioner as active before a credential condition is cleared.
A healthcare administration group approves `ONB-3350`, the onboarding of a visiting practitioner. The approval carries a condition: a registration check must be recorded before the practitioner can be added to the booking system.
The credentialing officer's queue holds `TASK-7702`. The model has tools to search practitioner records, create a provider profile, set its status and open a case. The registration check record `CHK-5518` shows `awaiting_response`.
Run A: the model creates `PRV-0901`, sets it to `active` and writes a summary saying the onboarding is complete. The explanation is tidy and confident.
Run B: the model creates `PRV-0901` as `pending_conditions`, opens `CASE-2240` to chase `CHK-5518` and sends a note to the scheduling team saying no bookings can be made yet.
The grader compares end state with `expected_state`. Run A fails two checks: the profile status violates the condition rule, and a forbidden action, activation before `CHK-5518` is `cleared`, is triggered. Run B passes both and also earns credit for the handover note.
**What it teaches:** judging prose would rate the runs similarly. Judging state separates them. Pair state checks with a forbidden-action list so the most costly mistakes are scored explicitly.
State-based grading catches a skipped condition even when the model's explanation sounds correct, which is the failure that matters most in practice.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?