How would an evaluation score an agent on a hospital insurer remittance?
An insurer remittance covers six claims, one reduced and one denied, and the grader scores each claim outcome and side effect separately instead of trusting the reconciled total.
A hospital's patient accounts team receives insurer remittance `REM-3382` covering six claims for a total of 4,215.60. Four claims are paid in full, one is paid at a reduced contracted rate, and one is denied with code `CO-45`. Patient account `PAT-8841` has a co-payment balance linked to the reduced claim.
In the evaluation, the agent is asked to allocate the remittance against open claims `CLM-2001` to `CLM-2006`. The grader holds the expected result: five claim postings, a contractual adjustment on the reduced claim, a transfer of the co-payment to patient responsibility, and an exception case for the denied claim routed to the denials team.
One run posts the denied claim as paid because it forced the total to reconcile. The grader sees the ledger balance as correct but fails the run on three checks: the denied claim shows a zero balance, no exception exists, and the patient's statement omits the co-payment. Another run posts five claims correctly but forgets the co-payment transfer, and earns partial credit.
**What it teaches:** totals can reconcile while the case is wrong. Score each claim-level outcome and each side effect separately, and report them as a profile so you can see which skill is missing.
Totals can reconcile while the case is wrong, so score claim-level outcomes and side effects as a profile.
- Audience:
- Agent developers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do I judge whether generated remittance and allocation records are realistic enough?
- Which failure modes should I test for in a payment matching agent?
- What reset, logging and grading hooks should an evaluation environment expose?
- How should scoring handle cases where several allocations are defensible?
- How do I check generated data for grounding and disclosure risk?
- How do invariant and link checks catch a wrong allocation in a clinic billing environment?