Skip to content
MirrorFirm

How should a task pack score more than one valid referral path?

Constraint-based scoring can recognise several acceptable action sequences while still rejecting invented promises, missing prerequisites and incomplete transfers in downstream evaluation runs.

**Must the model reproduce a reference answer exactly?**

No. A customer can receive a clear explanation in several forms, and independent preparation steps may occur in different orders. A single scripted answer can penalise valid behaviour while overlooking a fluent response that omits a required action.

**What should the scoring rubric contain?**

A task pack could identify mandatory prerequisites, prohibited actions and acceptable terminal states. It can separately assess whether the model accurately explains the next step and whether the required transfer actually occurs. Optional courtesy wording should not carry the same weight as correct routing.

**How can a buyer inspect ambiguous cases?**

Ask for the evidence supporting each scoring constraint and a review category for cases where the supplied policy does not resolve the decision. An evaluator should not manufacture a precise answer merely because the scoring system requires one.

**What does this mean for onward licensing?**

Resellers can distribute the rubric with the task definition so downstream customers know what a pass means. If customers replace the policy or tool permissions, they should revisit the constraints. A valid score belongs to a defined configuration, not to the wording of a reference response alone.

Avoid brittle answer matching by licensing rubrics that distinguish required outcomes from optional wording and action order.

Audience:
Licensed resellers
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Hardship referral