Skip to content
MirrorFirm

What are the success criteria and common failure modes when grading this workflow?

Answers on defining success from end state, the failure modes worth testing, how to score them separately, how to verify labels and how to probe for memorisation.

**What does success look like?** Define it per case from the final state, not the wording. The gate is open only when every mandatory condition is satisfied or validly waived; the right work item is released or held; the right role is notified; and the handover names each open condition.

**What are the common failure modes?** - Treating 'approved' as 'cleared'. - Accepting a document as evidence because it exists, without checking it meets the condition. - Missing a condition added after the original decision. - Proceeding on an expired approval. - Granting a waiver without the required authority. - Yielding to urgency or seniority in the request. - Holding correctly but giving the next person no usable handover.

**How should we score it?** Use separate checks per failure mode and report them individually, alongside an overall pass on end state. A single aggregate hides which behaviour broke.

**How do we tell whether the labels are right?** Have a domain reviewer read a sample of cases against the policy text and mark disagreements. Raise any label that depends on facts absent from the records.

**Can this detect memorisation?** Compare performance on cases with reworded or reordered records. A large gap suggests reliance on surface patterns.

Define success from final state, score each failure mode on its own, and verify labels with a domain reviewer before trusting results.

Audience:
Model labs
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Approval conditions