Skip to content
MirrorFirm

How does an evaluation catch a model that reschedules one delivery but not the dependent ones?

A logistics evaluation run in which a model moves a single delivery slot and misses a linked standing order, showing how step-level grading exposes a fault that a final-answer check would pass.

A task in a logistics pack starts with a distributor's delivery schedule, **DLV-5521**, for a retail customer. It holds six weekly drops, each with a booking in a carrier's slot table. Drop 3 is linked to a standing order, **SO-1180**, that releases stock to the warehouse two days earlier.

The customer service coordinator receives a request: the store is closed for fitting works in week 3, so move that drop by one week. Policy says a changed drop must keep a gap of at least five days from its neighbours, and any linked stock release moves with it.

**What the model does.** It updates the drop 3 date, sends a confirmation message to the customer, and reports success.

**What the grader finds.** The final message is fine. The state diff is not: the carrier booking for drop 3 still shows the old date, and SO-1180 will release stock on the old date, leaving it sitting in the warehouse for a week. The step log shows no call to the carrier booking tool.

**Scoring.** The task is marked partial: date change correct, notification correct, two linked records wrong. A grader that checked only the message would have passed it.

**What it teaches:** evaluate against linked-record state and tool-call logs, and report scores per step, so failures point to a specific gap in the model's behaviour.

Step-level grading and linked-record diffs expose faults that a final message check would miss, such as an unmoved dependent delivery.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Repayment scheduling