Should a correct final record receive full credit?
A correct final record can conceal unsupported actions or misleading customer communication, so evaluation should distinguish state correctness, execution conduct and completion claims.
**Is the final state enough to judge success?**
No. An agent can reach the expected value after editing an unrelated account, bypassing a required check or making an incorrect promise to the customer. A single final state score obscures those failures.
**What should be scored separately?**
Assess the requested change, unwanted side effects, compliance with the supplied action rules and the accuracy of the final message. A utilities agent may correctly submit a contact update but wrongly tell the customer that every dependent notice has already changed. That is a communication failure even if the queued work later succeeds.
**How should unresolved cases be treated?**
A justified pending outcome can be correct when required evidence or acknowledgement is absent. The evaluator should compare the agent's claim with the state at the moment it makes that claim. MirrorFirm could generate cases with separate expected state and communication criteria. Buyers can then distinguish an agent that stops appropriately from one that abandons available work, and distinguish honest partial completion from a confident but unsupported success message.
Separate scoring dimensions to reveal whether an agent reaches the right state for defensible reasons and describes it accurately.
- Audience:
- Environment builders
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?