Skip to content
MirrorFirm

How can buyers check whether an agent understands changes rather than memorising wording?

Paired tasks can vary request wording while preserving intent, or vary a decisive fact while preserving wording, to test whether agent behaviour follows meaning.

**What should stay the same across equivalent requests?**

When two generated requests express the same instruction against equivalent state, the agent should reach equivalent business outcomes. A formal message and a brief conversational message can both request removal of an optional retail subscription. Differences in politeness or sentence order should not change the result.

**What should cause behaviour to change?**

A decisive business fact should. The same request can arrive before or after a fulfilment commitment has been made. If the applicable rules differ at that boundary, an agent that produces identical actions in both cases may be ignoring state.

**How should these comparisons be designed?**

Ask for paired cases with a documented transformation and a declared expected relationship between outcomes. Change one relevant dimension at a time so a failed comparison has an interpretable cause.

These checks supplement individual task scoring. They reveal agents that pass familiar formulations but fail ordinary paraphrases, as well as agents that repeat a learned response when a changed business condition requires a different decision.

Use controlled task variations to separate language robustness from sensitivity to business conditions.

Audience:
Agent developers
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Customer change handling