How can evaluation hooks catch consequences that occur after the conversation ends?
Virtual time and scheduled event hooks let evaluators inspect delayed consequences, including whether an accepted referral actually changes downstream account activity as intended.
Add a virtual clock and an explicit event queue to the environment. A referral episode may finish before the next reminder, queue dispatch or account review occurs. Evaluating only at conversation closure misses whether the agent's actions changed those scheduled behaviours.
Generated event records can include `event_id`, `scheduled_at`, `event_type`, `target_id`, `guard_condition` and `execution_status`. A reminder event should inspect current account state when it executes, according to the configured policy, rather than rely solely on the state when it was scheduled.
Expose an evaluator-only action such as `advance_time` that processes due events in a documented order. Capture emitted notices and queue changes in an event trace. The acting agent should not receive private expectations attached to those events.
MirrorFirm could generate initial schedules and expected invariants for bounded time windows. A hook can test whether prohibited reminders remain suppressed while a relevant hold is active and whether unrelated permitted activity continues.
Reset must restore the clock, event queue and execution history together. Otherwise, a second run may inherit consumed events and appear to pass simply because the consequence was never tested again.
Evaluate downstream effects by advancing simulated time through pending events.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How should evaluation handle a customer changing information during referral submission?
- How can buyers test whether an agent understands referral triggers rather than keywords?
- How can a buyer test resistance to instructions hidden inside customer records?
- What should buyers inspect before using generated hardship material for training?
- Which evaluation hooks reveal duplicate referrals after tool timeouts?
- How do we check that the environment is not giving the agent the answer?