How do I judge realism and disclosure risk in a generated escalation sample?
Buyers can use domain reviewers, outcome and note variation checks, and their own matching for identifiers, treating disclosure review as their responsibility because generated output carries no promise of anonymity.
**How do I judge whether generated escalation data reflects how businesses really work?**
Ask domain staff on your side, or a trusted reviewer, to read a spread of cases and notes and mark anything that would not happen in practice: limits that are inverted, referrals that skip required levels without explanation, or notes that read like policy restatements. Disagreement is useful feedback for scoping.
**Can I test whether the output leaks anything sensitive?**
You can inspect for it, but you should not assume it is absent. The material is generated, yet generation can still produce text or identifiers that resemble real entities. Search for unusual specificity, run your own matching against names and identifiers you care about, and report anything suspicious before use. MirrorFirm does not claim that generated output is anonymous or compliant with any regime, so your own review is part of the process.
**What if the cases feel too tidy?**
Check outcome distribution and note variation. Tidy data often shows as uniform notes, evenly spaced amounts, or decisions predicted by one field. Ask for more messy cases such as missing evidence, borderline exposure and wrong-level referrals.
**Is one sample enough to decide?**
It is enough to judge structure and integrity. To judge variety, request a second sample with different seeds or a different domain and compare.
**Who decides what counts as passing?**
You do. Set the thresholds in terms of your own training or evaluation purpose.
Realism and disclosure risk are inspected, not assumed, and your own review is part of the process.
- Audience:
- Model labs
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- Which failure modes should an escalation evaluation cover?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?