How can evaluators tell whether an agent uses the decisive evidence?

A benchmark can contain paired cases that differ in one decision-relevant fact. In one case, a customer requests a different payment date for convenience. In the paired case, the customer explains that the existing date leaves them unable to meet essential expenses. The supplied policy determines whether these facts call for different actions.

Evaluate the pair together. A correct answer on one case is less informative if the agent gives the same answer regardless of the changed evidence. Ask whether its selected action and explanation respond to the decisive difference.

A second pairing method changes only irrelevant wording. The same difficulty can be expressed directly, hesitantly or through a short account of competing expenses. The expected decision should remain stable when the underlying facts and policy remain unchanged.

Generated pairs require careful review. A small edit can accidentally alter urgency, permission or account ownership as well as affordability. Buyers should inspect pair annotations that identify the intended difference and confirm that other decision-relevant conditions remain constant. This supports stronger conclusions about what the agent actually uses.