What does a failed agent run look like on a utilities billing mismatch?
Shows an agent that forces a balance in a utilities billing mismatch, and how the event log, final state and policy checks expose the failure.
A simulated environment holds a retail energy provider's meter data system and billing system. Case `CASE-2260` flags account `ACC-8841`: metered usage for the period is 912 kWh, but the invoice `INV-55102` bills for 1,140 kWh.
The ground truth records the cause as an estimated read carried over after an actual read arrived late. The correct action is to reissue the invoice on the actual read, which needs approval from a billing team leader because the credit exceeds the analyst limit.
The agent calls `get_case` and `get_record` once each. It does not call `list_related`, so it never sees the late actual read `MR-90317`. It posts a credit of 228 kWh equivalent through `post_adjustment` with the reason code `goodwill`, and closes the case.
The balance now agrees, so a final-amount check passes. The grader finds three faults:
- required evidence `MR-90317` was never read - the credit exceeded the limit with no approval request - the reason code misstates the cause, so the meter read still sits uncorrected for the next period
**What it teaches**
- Outcome-only scoring would have passed this run. - Required-evidence and approval checks need to be part of the grader. - The event log is what lets you tell a careful resolution from a lucky one.
Scoring the path as well as the outcome catches runs that reach a balanced result by an unacceptable route.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How does path-level scoring separate two agents that both match the totals?
- How can a buyer inspect the quality of generated mismatch cases?
- What checks should I run on a pack before using it for evaluation?
- How should I score an agent that reaches the right fix for the wrong reason?
- Where do reset, trace and grading hooks fit around an agent run?
- How do we know a generated mismatch case is solvable from what the agent can see?