Why should an evaluation distinguish policy validity from agent knowledge?
Separating effective time from publication and delivery time reveals whether an agent reasons correctly about what applied and what it could know.
A generated policy record can carry `policy_version_id`, `effective_from`, `effective_to` and `published_at`. A delivery event can separately record `recipient_role`, `available_at` and `acknowledged_at`. These fields represent different facts: when a rule applies, when it becomes accessible and when someone confirms receipt.
A utilities service request can link its decision event through `decided_at` and `policy_version_used`. The evaluator can then inspect whether the agent used the rule applicable to that event and whether the replacement rule was visible through its permitted tools at the time.
This distinction matters when a notice arrives after the effective time. A benchmark should not silently treat inaccessible information as something the agent ignored. It can instead test the response after discovery, such as identifying decisions that need authorised review.
JSONL event records can preserve both business timestamps and observation timestamps. An evaluation hook can reconstruct the policy view available at each action, rather than scoring against the final document collection. That makes the task a test of reasoning under changing information and prevents later knowledge from distorting the assessment of earlier behaviour.
Build temporal evaluations that distinguish late information, incorrect applicability decisions and unreasonable retrospective expectations.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- What it is and why it matters
Content release: r-a220eb6e099b243b
Related questions
- How can a policy change break a handover between connected operational teams?
- Why does a changed policy require more than replacing a reference document?
- Why should a policy task expose events as well as the latest record state?
- Should an agent apply every policy update uniformly to all open cases?
- What should developers map before an agent propagates a policy change?
- Which business features make this workflow worth simulating?