How can a lab test whether the model learned the workflow rather than memorised templates?
Evaluation splits can vary policy structure and dependency patterns, helping distinguish transferable workflow reasoning from familiarity with repeated language or identifiers.
**Is changing the wording enough to create a useful test set?**
No. A model can recognise the same dependency pattern beneath paraphrased messages. If every training case updates the same fields in the same order, a differently worded test may still reward a memorised procedure.
**What should a buyer ask to hold out?**
A lab can request splits by dependency structure, policy branch or relationship arrangement. Training tasks may involve a single service agreement, while a held-out family includes several agreements with different inheritance rules. The evaluation then tests whether the model reads the supplied structure instead of assuming the familiar one.
**How can the split itself be inspected?**
Ask for generation-family identifiers and a split manifest that keeps closely related variants together. Review whether answers, policy rationales or expected action sequences accidentally appear in model-visible inputs. Surface variation should be documented separately from structural variation.
MirrorFirm could generate matched evaluation families that isolate a particular transfer question. A lab can compare results across familiar and unfamiliar structures, while recognising that success within the generated distribution does not establish performance across every business setting.
Hold out business structures as well as wording to test whether training transfers to unfamiliar change tasks.
- Audience:
- Model labs
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- How can buyers evaluate correctness without requiring one exact action sequence?