How can an evaluation catch a model that reaches the right numbers by the wrong route?
In a utility billing case, a model overwrites an invoice figure so the numbers agree, and separate end-state, action and side-effect checks show where it went wrong.
A utility's billing team has a mismatch on account `ACC-60417`: the meter data system shows 412 kWh for the quarter, and invoice `BILL-33081` charges for 521 kWh. A lab runs a model as the billing analyst in a simulated environment and grades it on end state and on the action log.
The seeded cause is an estimated read that was never replaced when an actual read arrived. The meter data holds `READ-5512`, an actual read dated after the estimate, while billing still uses the estimate `READ-5498`. The correct action is to request a reversal and rebill using the actual read, with the reason code for superseded estimate, and to hold the overdue reminder.
The model under test overwrites the consumption figure on the invoice directly. The final numbers agree, so a state-only check would pass it. The log check fails it: policy says issued invoices change only through reversal and rebill, and the overdue reminder `CASE-1963` still fires.
The grader records three separate results: end-state agreement passed, action validity failed, downstream effects failed. The lab can see that the model reaches the right figure by the wrong route.
**What it teaches:** score the route and the side effects as well as the final numbers, and report them separately so weaknesses do not hide inside one pass rate.
Report end-state agreement, action validity and downstream effects as separate results so a model's real weaknesses are visible.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How does path-level scoring separate two agents that both match the totals?
- How can a buyer inspect the quality of generated mismatch cases?
- What checks should I run on a pack before using it for evaluation?
- How should I score an agent that reaches the right fix for the wrong reason?
- Where do reset, trace and grading hooks fit around an agent run?
- How do we know a generated mismatch case is solvable from what the agent can see?