Skip to content
MirrorFirm

How can a buyer judge the quality of generated set-up traces?

Quality can be judged on internal consistency, policy fidelity, realism of exceptions and whether the expected end state is reachable and checkable from the starting state.

Start with structure, then move to behaviour.

**Internal consistency.** Every identifier should resolve. Dates should run in a sensible order, statuses should follow permitted transitions, and amounts or limits should agree across records. Scripts can test most of this.

**Policy fidelity.** Take a sample of tasks and apply the stated policy by hand. The expected end state should follow from the rule, and the rule should be unambiguous enough that two reviewers reach the same answer.

**Exception realism.** Ask a person who has done this work whether the exceptions look like ones that occur: a mismatched address format, a condition cleared by the wrong document, a handover missing a reference. Unrealistic exceptions teach unrealistic caution.

**Reachability.** Confirm that a correct run can get from the starting state to the expected state using only the offered tools.

**Diversity.** Check whether tasks differ in substance or only in names and amounts. Many near-identical tasks inflate size without adding signal.

**Grounding and disclosure risk.** The material is generated, so inspect whether any record or text seems to mirror a real entity too closely, and ask how grounding was done.

No single check settles quality. Combine them, and agree pass criteria for the pilot before it starts.

These checks can be run on a sample before any commitment, and most do not require domain staff to review every record.

Audience:
Model labs
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Account setup