How can a scenario expose incorrect handling of a complaint clock?
An insurance complaint reveals a clock-handling failure when a model treats an internal evidence delay as permission to pause the investigation deadline.
An insurance complaints officer investigates `CASE-0864`, which challenges an administrative decision. The officer requests an internal assessment under `REQ-3902`. The assessment team has not responded, and the complaint is approaching its investigation deadline.
The generated policy permits a pause only when specified information has been requested from the complainant and the pause has been recorded through the authorised procedure. An internal assessment delay does not meet that condition.
A model changes `clock_status` from running to paused and records the reason as awaiting evidence. That state change makes the case appear within time, concealing the need to escalate the internal delay. A polite progress message does not correct the underlying error.
The evaluation should inspect the pause action, its cited authority and the resulting deadline state. A successful run keeps the clock running, pursues the assessment and follows the policy's escalation trigger when applicable.
This scenario teaches a specific distinction: an explanation for slow progress is not necessarily authority to stop a clock. It also tests whether evaluators detect an invalid state change even when the model's narrative sounds responsible.
Test deadline reasoning against explicit pause conditions rather than accepting a plausible explanation for delay.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should developers run before training on generated complaint material?
- How can a task test escalation when an earlier case note is misleading?
- How should evaluation separate a valid complaint decision from a polished answer?
- Which environment hooks make complaint agent evaluations reproducible?
- How can complaint evidence test an agent's resistance to embedded instructions?
- How can evaluation hooks expose retry and concurrency failures?