What failure modes should an intake evaluation measure?

**What failure modes matter most?** Four are worth separating: false completeness (routing a file that should be held), false incompleteness (holding a file that is complete), wrong routing (right check, wrong queue) and record damage (merging unrelated parties or closing the wrong case).

**Which is worse?** It depends on the business. In claims lodgement, false completeness pushes work onto assessors. In an urgent service request, false incompleteness delays a customer. A buyer can ask for task weighting that matches its own cost of error.

**How do we test escalation?** Include tasks where the correct action is to stop and hand over, such as a case that matches two policies with no tie-break. The expected outcome is an escalation note to a named role, not a guess.

**How do we catch agents that look right but act wrongly?** Grade state, not prose. Compare exported case records and the action log with the expected block, and include forbidden actions so a correct final status reached through a destructive step still fails.

**How do we avoid overfitting to the sample?** Hold back a test split with different case wording, party names and document layouts, and compare results between the splits. A large gap suggests the agent learned surface patterns of the training set.

**Is a high score proof the agent is ready for production?** No. It is evidence on generated tasks. Validate on your own process before relying on it.