How do I judge the quality of generated conditional approval data?
Quality shows up in consistency across linked records, fidelity to rule text, correct labels, realistic variety, and a careful look at grounding and disclosure risk.
Quality in conditional approval data shows up in whether the records stay consistent when you follow the links. Start with mechanics, then move to judgement.
**Internal consistency.** Evidence should be received after the condition exists. A review should follow the evidence it assesses. A released step should not precede the clearing of a blocking condition unless the case is built as a failure. Dates, roles and statuses should agree across files.
**Rule fidelity.** Read a handful of conditions and the evidence linked to each. Does the finding follow from the rule text? A threshold condition should be judged on the number, not on whether a document exists.
**Label correctness.** If a task carries an expected end state, check that a careful person reading the policy would reach it. Ambiguous cases should be marked as such, with an accepted set of outcomes.
**Realistic variety.** Look for different condition types, different evidence formats, late and partial submissions, and a spread of outcomes that includes holds. Data in which every case clears neatly will not separate strong agents from weak ones.
**Grounding and disclosure risk.** The output is generated, informed by observed practice. Ask how the generation was grounded, and check a sample for text that looks lifted from a real source or identifies a real party. Do not assume generated means free of that risk.
Run these checks on a small sample first, and record what fails.
Check mechanics first, then judgement, and treat grounding and disclosure risk as something to inspect, not assume away.
- Audience:
- Evaluation teams
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?