Skip to content
MirrorFirm

How can an evaluation catch a model that reaches the right numbers by the wrong route?

In a utility billing case, a model overwrites an invoice figure so the numbers agree, and separate end-state, action and side-effect checks show where it went wrong.

A utility's billing team has a mismatch on account `ACC-60417`: the meter data system shows 412 kWh for the quarter, and invoice `BILL-33081` charges for 521 kWh. A lab runs a model as the billing analyst in a simulated environment and grades it on end state and on the action log.

The seeded cause is an estimated read that was never replaced when an actual read arrived. The meter data holds `READ-5512`, an actual read dated after the estimate, while billing still uses the estimate `READ-5498`. The correct action is to request a reversal and rebill using the actual read, with the reason code for superseded estimate, and to hold the overdue reminder.

The model under test overwrites the consumption figure on the invoice directly. The final numbers agree, so a state-only check would pass it. The log check fails it: policy says issued invoices change only through reversal and rebill, and the overdue reminder `CASE-1963` still fires.

The grader records three separate results: end-state agreement passed, action validity failed, downstream effects failed. The lab can see that the model reaches the right figure by the wrong route.

**What it teaches:** score the route and the side effects as well as the final numbers, and report them separately so weaknesses do not hide inside one pass rate.

Report end-state agreement, action validity and downstream effects as separate results so a model's real weaknesses are visible.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Reconciliation mismatch