Skip to content
MirrorFirm

How do we check that the environment is not giving the agent the answer?

Leakage checks examine visible records, identifiers and tool responses for hidden outcome cues that let agents bypass the intended recognition and referral task.

**What counts as answer leakage here?**

Leakage occurs when an agent can infer the required action from information that should belong only to the evaluator. A visible field such as `expected_referral=true` is an obvious case. Less obvious cues include scenario filenames, identifiers grouped by outcome or tool messages that reveal the correct next action.

**How can a buyer inspect it?**

Request the exact observation payload shown to the agent, not just the underlying record bundle. Review rendered policy snippets, search results, error responses and task metadata. Private evaluation annotations must remain outside those surfaces.

**Can we test for indirect cues?**

Shuffle identifiers and neutralise non-business metadata while preserving the customer situation. If behaviour changes substantially, investigate whether the agent relied on an accidental signal. Also inspect whether positive and negative cases have different formatting or unusually consistent wording.

**What could MirrorFirm provide?**

A release can include a visibility map and paired versions with neutral metadata. These support a practical leakage audit, although passing that audit does not establish that every shortcut has been removed. Builders still need to inspect the complete runtime presentation produced by their own environment.

Audit the observation surface so success depends on business evidence rather than generator artefacts.

Audience:
Environment builders
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Hardship referral