How should scoring handle cases where several allocations are defensible?
Tasks can carry acceptance rules instead of one fixed answer, with partial credit by dimension and critical failures reserved for breaches of hard constraints.
**Why is scoring not just right or wrong?** Several outcomes can be acceptable. An invoice may be cleared oldest-first or by amount match, and both comply with a policy that permits either. Binary scoring would penalise a valid choice.
**How can a task pack handle that?** Each task can carry a set of acceptable allocations, or a rule such as "any split that clears items in due-date order", rather than one fixed answer. A grader tests the rule, not a string match.
**What deserves partial credit?** - Correct item selection but wrong amounts. - Correct postings but a missing follow-up case. - Correct escalation but a missing reason code.
**What should never earn credit?** Actions that breach a hard constraint: posting above an approval threshold without approval, applying cash to a disputed item against policy, or touching an account outside the payer's hierarchy. Score these as critical failures independent of the rest.
**How do I read the results?** Report per-dimension scores (selection, amount, side effects, escalation, constraint adherence) instead of one number. A single average hides an agent that is accurate but unsafe.
Ask for the grader source so you can adjust weights to your own risk tolerance.
Per-dimension scores and rule-based acceptance avoid penalising valid choices while still exposing agents that are accurate but unsafe.
- Audience:
- Agent developers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do I judge whether generated remittance and allocation records are realistic enough?
- Which failure modes should I test for in a payment matching agent?
- How would an evaluation score an agent on a hospital insurer remittance?
- What reset, logging and grading hooks should an evaluation environment expose?
- How do I check generated data for grounding and disclosure risk?
- How do invariant and link checks catch a wrong allocation in a clinic billing environment?