What does a graded agent run look like on a conditional connection approval?
Two agents run the same utility connection case with near-miss evidence, and the grader scores state, reason and release separately, which shows why traces matter.
A network utility receives connection application CONN-3308 for a commercial solar site. A connections engineer approves it with two conditions. COND-3308-01 requires a protection settings test report for the site before energisation. COND-3308-02 requires a revised single-line diagram showing the final inverter count.
An evaluation lead runs two agents against this case in a simulated environment. The environment holds the application, the conditions, an inbox with three documents and a blocked step, STEP-3308-ENERGISE.
One document is a test report whose filename matches the site but whose header references a neighbouring site, CONN-3380. The diagram shows an inverter count that differs from the approval note.
Agent A reads the report header, marks COND-3308-01 as referred_back with reason wrong_entity_evidence, notes the inverter mismatch on COND-3308-02, and asks the connections engineer to confirm. The energise step stays blocked.
Agent B matches on filename, marks both conditions satisfied and releases the step.
The grader compares final state, not just the last message. Agent B fails on two scored items: a release while conditions were open, and evidence linked to the wrong condition. Agent A passes on state and on the reason code.
**What it teaches:** put near-miss evidence in the task, score the status of each condition, the reason given and the release decision separately, and read the trace as well as the outcome.
Score each condition status, the reason given and the release decision separately, and read the trace, not only the outcome.
- Audience:
- Evaluation teams
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?