Skip to content
MirrorFirm

Why should an evaluation distinguish policy validity from agent knowledge?

Separating effective time from publication and delivery time reveals whether an agent reasons correctly about what applied and what it could know.

A generated policy record can carry `policy_version_id`, `effective_from`, `effective_to` and `published_at`. A delivery event can separately record `recipient_role`, `available_at` and `acknowledged_at`. These fields represent different facts: when a rule applies, when it becomes accessible and when someone confirms receipt.

A utilities service request can link its decision event through `decided_at` and `policy_version_used`. The evaluator can then inspect whether the agent used the rule applicable to that event and whether the replacement rule was visible through its permitted tools at the time.

This distinction matters when a notice arrives after the effective time. A benchmark should not silently treat inaccessible information as something the agent ignored. It can instead test the response after discovery, such as identifying decisions that need authorised review.

JSONL event records can preserve both business timestamps and observation timestamps. An evaluation hook can reconstruct the policy view available at each action, rather than scoring against the final document collection. That makes the task a test of reasoning under changing information and prevents later knowledge from distorting the assessment of earlier behaviour.

Build temporal evaluations that distinguish late information, incorrect applicability decisions and unreasonable retrospective expectations.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
What it is and why it matters

Content release: r-a220eb6e099b243b

More on Policy change propagation