How should customers score cases with more than one acceptable response?
Evaluation rubrics can recognise valid alternatives and justified deferrals, preventing resold task packs from rewarding a single arbitrary action sequence.
**Must an agent follow the reference action sequence exactly?**
Only where order is part of the supplied business rule. If independent evidence checks can occur in either order, both sequences should be acceptable. A reference trace is one completion, not automatically the full definition of success.
**Can requesting more evidence be a correct answer?**
Yes, when the task deliberately omits a fact needed for an authorised decision. The rubric should identify the missing fact and the appropriate recipient of the request. A vague request for more information should not receive the same credit as a targeted one.
**How can a reseller make these distinctions inspectable?**
Ask for outcome classes such as completed correction, justified escalation and justified deferral. Each class can include required evidence, forbidden actions and closure conditions. Customers can then understand why an unposted case may still be successful.
**What should remain a failure?**
Deferring a fully supported, authorised task without a relevant reason can be scored separately from a necessary pause. This prevents indiscriminate escalation from appearing equivalent to competent correction handling.
Help customers evaluate business correctness without overfitting agents to a reference transcript.
- Audience:
- Licensed resellers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can developers evaluate a correction beyond checking the final balance?
- What should buyers inspect before trusting a generated correction pack?
- How should an agent behave when a posting request times out?
- How do we distinguish a justified pause from an agent avoiding the task?
- How should evaluation hooks avoid leaking the expected correction to the agent?
- Which hooks reveal duplicate posting after a tool timeout?