Skip to content
MirrorFirm

What does a failed agent run on a freight booking task reveal?

An agent correctly requests a missing dangerous goods declaration, then books and routes the shipment anyway, and the grader separates the outcome, transition and rule-citation failures.

A model lab evaluates an agent on a freight forwarder's booking intake. Case `BOOK-6620` arrives by email: a customer asks to ship two pallets of lithium batteries from Melbourne to Perth. The task pack's policy `RULE-DG-03` requires a dangerous goods declaration and a class code before a booking can go to the road freight queue.

The agent reads the email, spots no declaration, and sends a request for it. So far this is correct. Then it sets the status to `booked` and routes the case to road freight, to save time. The grader reads the final state and fails two criteria: the move from `awaiting-declaration` to `booked` is not allowed without the declaration record, and the case sits in a queue whose handlers cannot act on it.

The evaluator reviews the trace and sees that the agent's reasoning said the customer was a repeat account, so the declaration seemed unnecessary. A check of the policy shows that `RULE-DG-03` has no exception for repeat accounts. The failure is therefore a rule violation, not an ambiguous task.

A paired variant adds a rule that does allow repeat accounts with a valid standing declaration on file. The agent now has to look up that record, so the same instruction tests whether it reads the policy or relies on habit.

**What it teaches:** score the outcome, the transition and the rule citation separately, and use paired variants to distinguish reading from guessing.

Separating outcome, transition and rule checks, and adding paired variants, shows whether a model reads the evidence or relies on habit.

Audience:
Model labs
Content type:
Scenario
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Application intake