What failure modes and evaluation questions should I plan for?

**How do I know a model solved it for the right reason?** Compare its stated cause and its cited evidence with the labelled root cause and evidence set. Score the evidence, not only the outcome.

**What failure modes should I look for?** - Forcing the numbers to agree by editing the wrong record - Accepting the first plausible cause - Acting beyond the role's authority - Ignoring downstream items such as notices or payments - Citing a record that does not exist - Escalating everything

**How do I detect over-escalation?** Include cases where the expected outcome is a correction within authority, and report unnecessary escalation separately from missed escalation.

**Can I check for memorisation?** Ask for a split by template and by identifier style, and hold out whole templates, so a model cannot rely on having seen the same structure in training.

**What if two correct outcomes exist?** A task pack can list accepted alternatives, and the grader can accept any one of them while still rejecting unsafe actions.

**Does passing mean a model is ready for a live finance team?** No. A generated evaluation tests specific behaviours. Readiness for production use needs its own assessment, with your own controls and review.