How do I check that generated tasks are hard enough and do not leak the answer?

**How do I know the tasks are hard enough?**

Run a simple baseline first, such as an agent that proceeds whenever the status says approved. If it scores well, the pack is too easy. Then try an agent that always waits. If that scores well, the pack lacks cleared cases. A useful pack punishes both.

**Can answers leak through the data?**

Yes, and it is worth testing. Check whether identifiers, field order, note wording or task phrasing reveal the right action. If every blocked case contains the word pending in the notes, an agent can learn the shortcut. Shuffle and rename fields, then see whether performance drops.

**What should I read by hand?**

Sample a spread of tasks across condition types and read the full record trail. Ask whether a practitioner would agree with the ground truth, and whether the policy text supports it.

**How do I handle ambiguous cases?**

Ask for ground truth with a reason code and a confidence flag, so ambiguous tasks can be graded separately or removed from headline scores.

**Is the data free of disclosure concerns?**

It is generated, but that does not remove the need to check. Look for unexpected resemblance to material you know, and review how the content was grounded. Raise anything doubtful before the release is used more widely.

**Should I split the data?**

Yes. Keep a held-out set for evaluation that is never used to tune prompts or train.