How does an evaluation catch a model that reschedules one delivery but not the dependent ones?
A logistics evaluation run in which a model moves a single delivery slot and misses a linked standing order, showing how step-level grading exposes a fault that a final-answer check would pass.
A task in a logistics pack starts with a distributor's delivery schedule, **DLV-5521**, for a retail customer. It holds six weekly drops, each with a booking in a carrier's slot table. Drop 3 is linked to a standing order, **SO-1180**, that releases stock to the warehouse two days earlier.
The customer service coordinator receives a request: the store is closed for fitting works in week 3, so move that drop by one week. Policy says a changed drop must keep a gap of at least five days from its neighbours, and any linked stock release moves with it.
**What the model does.** It updates the drop 3 date, sends a confirmation message to the customer, and reports success.
**What the grader finds.** The final message is fine. The state diff is not: the carrier booking for drop 3 still shows the old date, and SO-1180 will release stock on the old date, leaving it sitting in the warehouse for a week. The step log shows no call to the carrier booking tool.
**Scoring.** The task is marked partial: date change correct, notification correct, two linked records wrong. A grader that checked only the message would have passed it.
**What it teaches:** evaluate against linked-record state and tool-call logs, and report scores per step, so failures point to a specific gap in the model's behaviour.
Step-level grading and linked-record diffs expose faults that a final message check would miss, such as an unmoved dependent delivery.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?