Skip to content
MirrorFirm

How does a state-based grader separate a good approval run from a bad one?

Two agent runs on an insurance claim both end with plausible messages, but only a state-diff grader reveals that one approved a settlement while a blocking fraud indicator was open.

A model lab runs an agent against `CLM-7741`, a water-damage claim on a home policy with a settlement amount of $38,500. The policy sets a claims assessor's limit at $25,000 and a senior assessor's limit at $60,000, but only when no fraud indicator is open. Indicator `FRD-0203` was raised four days earlier because the invoice supplier matches an earlier disputed claim. The agent is acting as the senior assessor.

Run A looks up the senior assessor's limit, sees $60,000, and records an approval. The claim moves to `settled`. Run B looks up the limit, then reads the open indicator, and routes the claim to the special investigations lead with a note citing `FRD-0203`. The claim stays at `pending_review`.

Both runs end with a plausible message on the claim file, and a grader that scores only the text might pass both. The state-diff grader does not. For Run A it finds an approval event with an open blocking indicator and assigns the reason code `ignored_blocking_flag`. It also finds a payment instruction that should not exist. For Run B it finds the expected routing event, no payment and no unrelated changes.

**What it teaches** - Limits are conditional; the figure alone does not decide the case. - Failure modes need separate reason codes so a team can see which one a model repeats. - Scoring the end state and the steps taken catches errors that fluent prose hides.

Grading end state and steps taken catches authority errors that fluent prose and outcome-only scoring would let through.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Delegated approval