Skip to content
MirrorFirm

How should an environment represent referral progress rather than just conversation quality?

Explicit referral states let evaluators distinguish an offer, a submitted request and specialist acceptance, preventing conversational fluency from standing in for operational completion.

A generated environment can represent referral progress as a state machine linked to the customer interaction. Useful fields include `interaction_id`, `case_id`, `referral_id`, `referral_state` and `state_reason`. Define each state before building tasks: an offer is different from a submitted request, and submission is different from specialist acceptance.

Tool actions could include `offer_referral`, `submit_referral` and `read_referral_status`. Each action should specify its preconditions and resulting state. If specialist acceptance happens through an environment event, the agent should not be able to manufacture that event by changing a text field.

Evaluation hooks can compare the final customer message with the last confirmed state. “Your request has been submitted” may be accurate while “the specialist team has accepted your case” is not. The wording matters because it communicates operational progress.

Store transitions as ordered JSONL events and retain a separate final snapshot for inspection. Reset should restore the opening state and remove transitions from the previous run. This structure lets evaluators distinguish a correctly described incomplete referral from an unsupported claim of completion.

Score observable progress through a defined process instead of inferring completion from the agent's final message.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
What it is and why it matters

Content release: r-a220eb6e099b243b

More on Hardship referral