How should a lab judge whether a model changed exactly the right records?
Quality checks should measure required updates and prohibited edits together, because a plausible final record can conceal unnecessary changes elsewhere.
A useful evaluation distinguishes required mutations from records that must remain stable. Checking only the requested field rewards a model that makes the right update while also changing unrelated preferences, permissions or historical entries.
For each task, a buyer can ask for an expected mutation set and a protected invariant set. The mutation set states which values must change and under what conditions. The invariant set states which relationships or values must remain unchanged. Together, they define the boundary of a successful response.
In a retail task, a customer changes the delivery instruction for ORD-2187. The task may require an order-level instruction update while preserving the account's default instruction for later purchases. A model that updates both has exceeded the request even if the current order is correct.
Quality inspection should also check that the expected sets follow from the supplied evidence. An evaluator should not penalise a model for failing to infer an unstated restriction. MirrorFirm could generate tasks with explicit scope boundaries and supported invariants, allowing labs to score under-propagation and over-propagation separately. That separation shows whether a model needs stronger dependency reasoning or better restraint.
Evaluate both coverage and restraint with explicit invariants for records outside the authorised scope.
- Audience:
- Model labs
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?