How should an environment represent referral progress rather than just conversation quality?
Explicit referral states let evaluators distinguish an offer, a submitted request and specialist acceptance, preventing conversational fluency from standing in for operational completion.
A generated environment can represent referral progress as a state machine linked to the customer interaction. Useful fields include `interaction_id`, `case_id`, `referral_id`, `referral_state` and `state_reason`. Define each state before building tasks: an offer is different from a submitted request, and submission is different from specialist acceptance.
Tool actions could include `offer_referral`, `submit_referral` and `read_referral_status`. Each action should specify its preconditions and resulting state. If specialist acceptance happens through an environment event, the agent should not be able to manufacture that event by changing a text field.
Evaluation hooks can compare the final customer message with the last confirmed state. “Your request has been submitted” may be accurate while “the specialist team has accepted your case” is not. The wording matters because it communicates operational progress.
Store transitions as ordered JSONL events and retain a separate final snapshot for inspection. Reset should restore the opening state and remove transitions from the previous run. This structure lets evaluators distinguish a correctly described incomplete referral from an unsupported claim of completion.
Score observable progress through a defined process instead of inferring completion from the agent's final message.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- What it is and why it matters
Content release: r-a220eb6e099b243b
Related questions
- What happens when a support request arrives while a utility service action is already queued?
- What should developers map before automating a hardship intake workflow?
- Why must an agent recognise financial difficulty before continuing collections?
- Should a general service agent decide which hardship assistance a customer receives?
- How should a connected agent represent a referral separately from a conversation?
- What state model makes this workflow more than a scripted conversation?