Skip to content
MirrorFirm

How should I score an agent that reaches the right fix for the wrong reason?

Explains separating cause, action, evidence, routing and note scores so lucky passes are exposed, and how to handle accepted equivalents and model-based grading of notes.

**How do I score a right fix reached for the wrong reason?** Score the cause and the action separately. A single pass or fail hides it.

**What are the components?** - **Cause:** does the agent's stated cause match the labelled root cause or an accepted equivalent? - **Action:** did it take an allowed action and avoid every forbidden one? - **Evidence:** did it read the records needed to support its cause before acting? - **Routing:** if approval or handover was required, was it sent to the right role? - **Note:** does the case note contain the required elements, such as cause, records cited and next step?

**What does a lucky pass look like?** The final state is correct but the agent never read the record that explains the variance, or its note names a cause that does not match the label. Evidence and cause scores will expose this.

**What about cases with more than one acceptable cause?** Ask for an accepted-equivalents list per case, so reasonable wording is not marked wrong.

**Can I use a model as grader?** You can for the note, with a rubric derived from the expected elements. Keep state and action checks deterministic, and spot-check the model grader against human review before you trust it.

Component scores expose lucky passes that a single pass or fail would hide, and show which part of your agent actually improved.

Audience:
Agent developers
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Reconciliation mismatch