How should a task pack score more than one valid referral path?
Constraint-based scoring can recognise several acceptable action sequences while still rejecting invented promises, missing prerequisites and incomplete transfers in downstream evaluation runs.
**Must the model reproduce a reference answer exactly?**
No. A customer can receive a clear explanation in several forms, and independent preparation steps may occur in different orders. A single scripted answer can penalise valid behaviour while overlooking a fluent response that omits a required action.
**What should the scoring rubric contain?**
A task pack could identify mandatory prerequisites, prohibited actions and acceptable terminal states. It can separately assess whether the model accurately explains the next step and whether the required transfer actually occurs. Optional courtesy wording should not carry the same weight as correct routing.
**How can a buyer inspect ambiguous cases?**
Ask for the evidence supporting each scoring constraint and a review category for cases where the supplied policy does not resolve the decision. An evaluator should not manufacture a precise answer merely because the scoring system requires one.
**What does this mean for onward licensing?**
Resellers can distribute the rubric with the task definition so downstream customers know what a pass means. If customers replace the policy or tool permissions, they should revisit the constraints. A valid score belongs to a defined configuration, not to the wording of a reference response alone.
Avoid brittle answer matching by licensing rubrics that distinguish required outcomes from optional wording and action order.
- Audience:
- Licensed resellers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How should evaluation handle a customer changing information during referral submission?
- How can buyers test whether an agent understands referral triggers rather than keywords?
- How can a buyer test resistance to instructions hidden inside customer records?
- What should buyers inspect before using generated hardship material for training?
- Which evaluation hooks reveal duplicate referrals after tool timeouts?
- How do we check that the environment is not giving the agent the answer?