How can buyers tell whether an agent follows policy instead of superficial cues?
Paired cases change one decision-relevant fact while preserving surrounding context, revealing whether an agent responds to the condition that actually controls follow-up.
An agent can appear competent by learning that old invoices usually need reminders. A stronger evaluation checks whether its action changes when the policy-relevant evidence changes, even though the invoice age remains identical.
Use paired cases with a controlled difference. One case can contain an authorised contact restriction and the other an expired restriction. The expected action changes because the restriction changes. A separate pair can alter an irrelevant presentation detail, such as the ordering of account notes, while keeping the expected action stable.
MirrorFirm could generate these pairs with a manifest describing the changed field, the governing rule and the expected effect on the decision. Keep that manifest outside the agent's observation surface so it supports scoring without revealing the answer.
Evaluate both kinds of behaviour: responsiveness to meaningful changes and resistance to irrelevant ones. Review the agent's cited evidence as well as its final action. A correct outcome reached through an unrelated rationale may fail when the same policy condition appears in a different business context.
Measure sensitivity to controlling evidence and stability against irrelevant changes within the same evaluation design.
- Audience:
- Agent developers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should buyers run before using a generated task pack?
- Which evaluation hooks expose duplicate actions and unreliable recovery?
- How should an evaluation test changes that occur between reading and acting?
- How should follow-up quality be scored when several responses are acceptable?
- What should we inspect before trusting an environment's evaluation results?
- How can we tell whether an agent learned the policy or a dataset shortcut?