What success criteria and failure modes should I use for intake tasks?

**What success criteria suit an intake task?**

Use several, scored separately: correct final status, correct queue and owner, required requests created, protected fields unchanged, and a handover note that names the open issues.

**Should the final state be the only check?**

No. Two runs can reach the same state by different paths, one safe and one not. Add trace checks such as whether a duplicate search was run before a case was created or merged.

**What are common failure modes?**

- Routing before completeness is confirmed - Overwriting a field instead of raising a discrepancy - Closing a request when only part of it was supplied - Creating a duplicate party or case - Applying a written rule without its exception - Attempting invalid transitions that the environment should reject

**How do I handle ambiguous cases?**

Ask for tasks where policy and practice differ, and decide in advance which behaviour you will reward. Record that choice in the grader so results can be compared later.

**What evaluation questions are worth asking?**

Does the score fall when you remove the policy text? Does it change if tool names are reordered? Does the agent recover after a rejected action? Each question tests a different weakness.

**Are the tasks themselves checked?**

Run a human or scripted reference solution on a sample, because a task that cannot be solved is a pack defect.