Skip to content
MirrorFirm

How should scoring handle cases where several allocations are defensible?

Tasks can carry acceptance rules instead of one fixed answer, with partial credit by dimension and critical failures reserved for breaches of hard constraints.

**Why is scoring not just right or wrong?** Several outcomes can be acceptable. An invoice may be cleared oldest-first or by amount match, and both comply with a policy that permits either. Binary scoring would penalise a valid choice.

**How can a task pack handle that?** Each task can carry a set of acceptable allocations, or a rule such as "any split that clears items in due-date order", rather than one fixed answer. A grader tests the rule, not a string match.

**What deserves partial credit?** - Correct item selection but wrong amounts. - Correct postings but a missing follow-up case. - Correct escalation but a missing reason code.

**What should never earn credit?** Actions that breach a hard constraint: posting above an approval threshold without approval, applying cash to a disputed item against policy, or touching an account outside the payer's hierarchy. Score these as critical failures independent of the rest.

**How do I read the results?** Report per-dimension scores (selection, amount, side effects, escalation, constraint adherence) instead of one number. A single average hides an agent that is accurate but unsafe.

Ask for the grader source so you can adjust weights to your own risk tolerance.

Per-dimension scores and rule-based acceptance avoid penalising valid choices while still exposing agents that are accurate but unsafe.

Audience:
Agent developers
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Payment allocation