Skip to content
MirrorFirm

How should a lab judge whether a model changed exactly the right records?

Quality checks should measure required updates and prohibited edits together, because a plausible final record can conceal unnecessary changes elsewhere.

A useful evaluation distinguishes required mutations from records that must remain stable. Checking only the requested field rewards a model that makes the right update while also changing unrelated preferences, permissions or historical entries.

For each task, a buyer can ask for an expected mutation set and a protected invariant set. The mutation set states which values must change and under what conditions. The invariant set states which relationships or values must remain unchanged. Together, they define the boundary of a successful response.

In a retail task, a customer changes the delivery instruction for ORD-2187. The task may require an order-level instruction update while preserving the account's default instruction for later purchases. A model that updates both has exceeded the request even if the current order is correct.

Quality inspection should also check that the expected sets follow from the supplied evidence. An evaluator should not penalise a model for failing to infer an unstated restriction. MirrorFirm could generate tasks with explicit scope boundaries and supported invariants, allowing labs to score under-propagation and over-propagation separately. That separation shows whether a model needs stronger dependency reasoning or better restraint.

Evaluate both coverage and restraint with explicit invariants for records outside the authorised scope.

Audience:
Model labs
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Customer change handling