Skip to content
MirrorFirm

How should an environment represent when a model could know about a rule change?

Separating policy validity from information visibility lets an evaluation distinguish poor reasoning from a decision made before the relevant update became accessible.

An environment can represent two separate clocks: when a rule applies in the business and when evidence of that rule becomes visible to the agent. Combining them can make an evaluation unfair or allow information from the future to enter a training example.

A generated event record could contain `event_id`, `occurred_at`, `visible_at`, `policy_version_id` and `record_id`. Policy state can separately contain `effective_from`. The observation interface should return only events whose `visible_at` value has been reached, even when the underlying simulator already holds later events.

This matters when a logistics dispatch rule takes effect while an agent is working from a delayed synchronisation feed. A decision made before the update becomes accessible should be assessed using the task's stated expectations for refreshing information and checking current authority. It should not be judged using hidden knowledge without an explicit requirement to obtain it.

JSONL event streams can preserve this distinction for replay. An evaluation hook can capture the observation snapshot alongside each action. Buyers can then inspect whether a failure came from ignoring visible evidence, failing a required refresh, or receiving insufficient information.

Measure decisions against the information available to the agent at the moment of action.

Audience:
Model labs
Content type:
Integration
Buyer question:
What it is and why it matters

Content release: r-a220eb6e099b243b

More on Policy change propagation