Skip to content
MirrorFirm

How can a lab test whether the model learned the workflow rather than memorised templates?

Evaluation splits can vary policy structure and dependency patterns, helping distinguish transferable workflow reasoning from familiarity with repeated language or identifiers.

**Is changing the wording enough to create a useful test set?**

No. A model can recognise the same dependency pattern beneath paraphrased messages. If every training case updates the same fields in the same order, a differently worded test may still reward a memorised procedure.

**What should a buyer ask to hold out?**

A lab can request splits by dependency structure, policy branch or relationship arrangement. Training tasks may involve a single service agreement, while a held-out family includes several agreements with different inheritance rules. The evaluation then tests whether the model reads the supplied structure instead of assuming the familiar one.

**How can the split itself be inspected?**

Ask for generation-family identifiers and a split manifest that keeps closely related variants together. Review whether answers, policy rationales or expected action sequences accidentally appear in model-visible inputs. Surface variation should be documented separately from structural variation.

MirrorFirm could generate matched evaluation families that isolate a particular transfer question. A lab can compare results across familiar and unfamiliar structures, while recognising that success within the generated distribution does not establish performance across every business setting.

Hold out business structures as well as wording to test whether training transfers to unfamiliar change tasks.

Audience:
Model labs
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Customer change handling