Which failure modes should I test for in intake tasks, and how do I detect them?

**What failure modes should I test for?** - **Premature routing:** sending a case onward before completeness rules are met. - **Over-asking:** requesting documents the policy does not require, which slows the applicant and the queue. - **Wrong merge:** treating two legitimate requests as duplicates, or missing a true duplicate. - **Identifier repair errors:** linking a submission to the wrong account after a near match. - **Rule drift:** applying an old policy version to a case lodged after a change. - **Thin notes:** leaving a handover note that a colleague cannot act on. - **Unsupported confidence:** closing a case when evidence is ambiguous instead of escalating.

**How do I detect them?** Compare the final state to the reference, check each status transition against the allowed list, and score the note for named items and cited rules.

**How do I tell a model failure from a data failure?** Run several models or settings on the same task. If strong runs all fail the same way, review the task for ambiguity or an error in the reference.

**Should I include cases where the right answer is to escalate?** Yes. Ask the pilot to include tasks where escalation to a named approver is the expected outcome, and score restraint as well as action.

All material is generated, so also review a sample for unintended resemblance to real entities.