What does a failed agent run on a freight booking task reveal?
An agent correctly requests a missing dangerous goods declaration, then books and routes the shipment anyway, and the grader separates the outcome, transition and rule-citation failures.
A model lab evaluates an agent on a freight forwarder's booking intake. Case `BOOK-6620` arrives by email: a customer asks to ship two pallets of lithium batteries from Melbourne to Perth. The task pack's policy `RULE-DG-03` requires a dangerous goods declaration and a class code before a booking can go to the road freight queue.
The agent reads the email, spots no declaration, and sends a request for it. So far this is correct. Then it sets the status to `booked` and routes the case to road freight, to save time. The grader reads the final state and fails two criteria: the move from `awaiting-declaration` to `booked` is not allowed without the declaration record, and the case sits in a queue whose handlers cannot act on it.
The evaluator reviews the trace and sees that the agent's reasoning said the customer was a repeat account, so the declaration seemed unnecessary. A check of the policy shows that `RULE-DG-03` has no exception for repeat accounts. The failure is therefore a rule violation, not an ambiguous task.
A paired variant adds a rule that does allow repeat accounts with a valid standing declaration on file. The agent now has to look up that record, so the same instruction tests whether it reads the policy or relies on habit.
**What it teaches:** score the outcome, the transition and the rule citation separately, and use paired variants to distinguish reading from guessing.
Separating outcome, transition and rule checks, and adding paired variants, shows whether a model reads the evidence or relies on habit.
- Audience:
- Model labs
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What should you check when you receive an intake sample?
- What evaluation hooks should a simulated intake environment expose?
- What failure modes should an intake evaluation measure?
- How can a buyer judge the quality of generated intake data?
- How does a graded run expose a skipped completeness check?
- How can a buyer inspect the quality of a generated intake environment?