How can we tell whether an agent follows policy rather than familiar wording?
Matched review cases can vary one policy-relevant fact while holding surrounding material stable, exposing reliance on superficial cues or memorised response patterns.
**How can we tell whether an agent follows policy?**
Use matched cases in which a single decisive condition changes. A scheduled logistics-carrier review might require escalation when an unresolved safety finding is classified as critical. A paired case can change the classification to advisory while preserving the surrounding records.
**What should change in the response?**
The required action should change only where the policy says it should. The agent should identify the altered condition and explain its effect. A different tone or longer summary is not evidence of policy sensitivity.
**What other pairings help?**
A second family can vary irrelevant details, such as document order or record identifiers, while preserving the expected outcome. These cases test stability when presentation changes but the decision basis does not.
**What should we inspect?**
Ask for a documented change ledger for every generated pair. Reviewers should verify that no accidental difference introduces another valid reason for the outcome. MirrorFirm could generate these paired cases for evaluation or training, with related variants kept together when assigning dataset splits.
Use controlled case pairs to test whether decisions respond to the facts and rules that should determine them.
- Audience:
- Evaluation teams
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should precede training on generated review cases?
- How can a late event reveal whether an agent checks its assumptions before committing a decision?
- Which evaluation hooks make review runs reproducible and auditable?
- How should evaluation distinguish a sound review decision from a persuasive explanation?
- How can we tell whether an agent responds to meaningful evidence rather than superficial cues?
- How should buyers inspect realism and disclosure risk in generated review material?