Skip to content
MirrorFirm

How can buyers check that an agent learned the workflow rather than the answer pattern?

Evaluation splits should separate related case families and include counterfactual policy changes, reducing the value of memorised surface patterns in follow-up decisions.

**How can buyers detect template memorisation?**

Start by identifying shared generation ancestry. Cases derived from the same base narrative, policy fragment or record graph should not be scattered casually across training and evaluation splits. Different identifiers do not make closely related cases independent.

**What is a useful behavioural check?**

Create contrast pairs where the surface wording stays similar but a decisive policy condition changes. A response request may be eligible for escalation in one case and require approval in the other. The agent should change its action and cite the changed condition.

Also test the reverse: vary message wording and record ordering while preserving the correct decision. An agent that changes its answer under irrelevant presentation changes may be relying on fragile cues.

**What could MirrorFirm provide?**

A task pack could include family identifiers, split assignments and documented transformation rules. Evaluator labels and rationales should remain outside the agent's input view.

Buyers should inspect both decision consistency and explanation fidelity. A correct answer paired with a rationale that cites an unchanged feature is weak evidence of policy understanding. These checks help separate genuine transfer across business cases from successful recognition of recurring generation patterns.

Use controlled contrast cases to test whether decisions follow policy evidence instead of familiar wording or record identifiers.

Audience:
Evaluation teams
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Overdue follow up