What does a first sample request and review look like for an environment lead?
An environment lead specifies a trade-account escalation sample, loads it, runs a reference agent and finds a tool-contract mismatch that shapes the next iteration.
An environment lead at an agent-evaluation team prepares sample request SR-0044 for a retail operations setting: a wholesaler's credit controller who may raise trade account limits up to 15,000 and must escalate larger increases to a finance manager.
The request states the domain, the single amount-based limit, three target failure modes (over-authority approval, limit avoidance through staged increases, and thin handover notes), and the team's tool contract: `get_account`, `raise_limit`, `escalate` and `add_note`. It asks for JSONL tables, a schema file and a task file with expected end states.
The sample arrives and the lead loads it into the environment. The request moves from DRAFT to SAMPLE_REVIEWED once the checks run. A scripted reference agent passes the within-authority tasks, but fails the staged-increase task: the data records two increases against ACCT-8812 as separate events with no shared key, so the grader cannot aggregate them.
**Consequence:** the lead feeds this back as a change request for a `limit_request_group_id` field. The next iteration adds the link and the reference agent passes.
**What it teaches:** - Run a reference agent before judging any model. - Aggregation keys are easy to omit and expensive to discover late. - The review produces specific, actionable requests, which is the purpose of a first step.
A sample reviewed against a reference run surfaces interface mismatches early, before they are baked into a larger release.
- Audience:
- Environment builders
- Content type:
- Scenario
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What licensing questions should I raise for generated escalation material?
- What might a first escalation pilot show an agent team?
- What should my team prepare to start an escalation pilot?
- How should I scope a first escalation pilot with MirrorFirm?
- What should I ask for in an escalation sample or pilot?
- What licensing, limits and boundaries should a buyer ask about before a pilot?