How should buyers score whether the right records changed?
Evaluation should score required updates, forbidden updates and justified deferrals separately, exposing systems that achieve completion by changing too much.
A final-state check should distinguish three categories: records that must change, records that must stay unchanged and records whose update depends on an unresolved condition. Scoring only the first category rewards agents that spread a customer instruction too widely.
In healthcare administration, a patient changes the contact number for appointment reminders. The requested update may apply to reminder preferences while leaving an emergency contact untouched. An agent that copies the number into both fields has completed the visible request and introduced a separate error.
A generated evaluation pack can provide assertions for required values, preserved values and permitted pending states. Each assertion should identify the supporting request or policy clause. This makes failures inspectable and helps downstream buyers distinguish misunderstanding from an incomplete action sequence.
For resellers, report these dimensions separately rather than hiding them inside one success label. A customer may care especially about unintended mutations, while another is studying unnecessary deferrals. Include the treatment of partially completed tasks in the scoring documentation. A useful evaluation contract explains both what counts as completion and what collateral changes invalidate an otherwise correct result.
Resellers can provide an evaluation contract that rewards precise execution and protects records outside the request's scope.
- Audience:
- Licensed resellers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?