How can I inspect the quality of an escalation dataset before licensing it onward?
Inspect internal consistency, referential integrity, threshold coverage, text quality, split hygiene and disclosure risk, since each affects whether your customers can rely on the release.
Start with checks you can run yourself on a sample.
**Consistency.** Every escalated case should have a requested amount or condition that exceeds the reviewing role's limit on the decision date. Cases that did not escalate should sit within it. Mismatches in either direction show weak generation.
**Referential integrity.** Each event should join to a case, each case to a reviewer, each decision to a rule version. Orphans are a warning sign.
**Threshold coverage.** Look at cases just below, at and just above each limit. A release with only obvious cases will not test boundary judgement.
**Text against fields.** Read handover notes beside the structured record. A rationale citing a figure that differs from the record is a defect.
**Variety.** Check that rationales are not near-copies of each other and that outcomes branch across approval, decline and return with conditions.
**Split hygiene.** Confirm all events for a case sit in one split.
**Grounding and disclosure.** The output is generated, so ask how it was produced and look for unusual detail that could point to a real source. Passing these checks does not remove disclosure risk, so keep your own review in the process.
Inspecting a release before listing protects your reputation, because the flaws your customers find first are usually checkable in advance.
- Audience:
- Licensed resellers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do evaluation hooks and environment resets work for escalation runs?
- Which failure modes should an escalation evaluation cover?
- How can I judge the quality of generated escalation cases?
- What success criteria should I score for an escalation task?
- How does an evaluation run expose an agent that escalates to the wrong reviewer?
- How would an evaluation catch an agent splitting a claim to stay under its limit?