How should evaluation distinguish complete handling from a correct-looking final record?
Evaluation can test business invariants across a change, detecting partial completion and damaging side effects that a final customer record alone cannot reveal.
A final field comparison answers whether the requested value appears in one place. It does not establish whether the business can continue operating correctly. Evaluation should also inspect the constraints that must hold after the agent's actions.
In a professional services account, changing the billing entity may require future unissued charges to reference the new entity while issued invoices remain attached to their original issuer and recipient. A task can therefore have both positive obligations and preservation constraints. Updating every invoice would satisfy a crude text check while violating the business rule.
A buyer can ask for an outcome rubric that separates request fulfilment, dependent record consistency and unintended side effects. The evaluator can report each dimension independently, making it possible to distinguish an incomplete but contained attempt from a completed change that damaged historical records.
This approach also supports useful training feedback. Instead of saying only that a task failed, the evaluation can identify which invariant broke and which action caused it, giving developers a concrete basis for improving agent behaviour.
Define success through required outcomes and preserved constraints across the affected process.
- Audience:
- Agent developers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How can buyers evaluate correctness without requiring one exact action sequence?
- What should buyers inspect before trusting a generated evaluation pack?