How do we score answers and what failure modes should we expect?

**How do we score an answer?** Use several measures per case: cause label match, action match against the permitted set, final record state, and whether escalation was appropriate. Report them separately, because a single score hides which part failed.

**What failure modes should we expect?** Common ones include forcing the totals to agree, editing the correct record, picking a cause from a surface pattern, ignoring the policy, inventing evidence, and escalating everything.

**How do we check for overfitting?** Hold back cases by cause type and by domain, and check performance on causes the model was not tuned on. Ask for a release split that supports this.

**How do we know the labels are right?** Review a sample by hand, ideally by someone who has done reconciliation work. Check that each label explains the gap and that the expected action is allowed by the attached policy.

**Can the data train a model as well as test it?** A buyer can ask for separate training and evaluation partitions, so evaluation cases are not seen during tuning. Confirm the licence allows both uses.

**Is passing proof of production readiness?** No. Passing shows performance on generated cases. It does not show behaviour on a customer's own systems, so test again against their data before deployment.

**Does the data carry privacy risk?** It is generated, but inspect grounding and disclosure risk rather than assuming there is none.