How does an end-state check catch a wrong policy start date?
An insurance scenario shows an agent that creates a policy with the wrong effective date, and how a state comparison and an ordering check expose the error.
An underwriter approves quote `QTE-6203` for a small business package policy, effective on the date the premium payment clears, not the date of approval. The policy administration team receives the work as `SETUP-1187`.
The agent creates policy `POL-33018` and sets the effective date to the approval date. Every tool call succeeds. The agent's summary reads cleanly: policy created, documents queued, customer notified.
The evaluation compares the final state with the expected state:
- `effective_date` differs from the expected value, which depends on payment record `PAY-8840`. - The certificate of currency was issued before payment cleared, an ordering violation. - The customer notice references a start date that is now wrong.
A score based only on whether a policy exists would pass this run. A field-level diff fails it, and the ordering check shows why: the agent never read the payment record before setting the date.
**What it teaches**
- Success of a tool call is not evidence of a correct outcome. - Dependencies between records should appear in the expected state. - Side effects such as certificates and notices should be scored, because they are hard to reverse. - Keep process checks and outcome checks separate so that you can tell a lucky pass from a sound one.
End-state diffs find errors that a successful tool call and a fluent summary would hide.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?