How can a grader catch a quiet error in a premium instalment change?
A scenario where an insurer's agent changes a premium instalment plan correctly on the surface, and a state-diff grader finds an unreversed late fee that a text check would miss.
A home insurer's environment contains policy `POL-2204` on monthly premium instalments. The March debit failed, a late fee `FEE-871` was applied, and the customer asks through `CASE-1175` to move the due date to the 20th. Company policy says the fee must be waived once, on first failure, if the date change is accepted.
**The agent's run:** it reads the case, moves the due date by creating schedule version 3, updates the debit instruction and writes a polite note saying the change is complete. A reviewer reading the transcript would call it a success.
**What the grader checks against state**
- schedule version 3 active, version 2 superseded: pass - instruction date matches version 3: pass - `FEE-871` status should be `waived` with reason code `first_failure`: **fail**, it is still `charged` - case note cites the waiver: **fail**
The state diff shows the fee untouched. Left alone, it would appear on the next statement and prompt a complaint.
**What it teaches**
- Grade the resulting state, not the agent's description of it. - Include policy-driven side effects in the expected end state. - Report failures by check, so a team can see that the agent handles dates well but misses fee rules, and target training data at that gap.
State-diff grading finds side-effect errors that look fine in the agent's own summary, which is where schedule agents most often fail.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?
- Which failure modes should schedule tasks be designed to expose?