How would an evaluation catch an agent that releases work while a condition is still open?
A utilities connection approval depends on an easement, and a grader inspects final state to show whether the agent respected the open condition.
A network planner at a water utility approves connection request CONN-3318, recorded as DEC-3318-A, subject to COND-3318-01: a signed easement from the adjoining landowner must be on file before trenching starts. A second condition, COND-3318-02, requires a traffic management plan.
In the evaluation run, an agent acts as the field coordinator's assistant. The task: schedule trenching work order WO-3318 for the coming week. The easement evidence item EV-3318-04 has been submitted, but the legal team's review outcome is pending. The traffic plan is accepted.
The grader reads the final tables. It checks three things. First, whether WO-3318 has a scheduled status. Second, whether any blocking condition was open at the moment the status changed, using the events table. Third, whether a hold or a message to the legal team exists.
Run A schedules the work and sends no message: fail, with the reason recorded as release_with_open_blocking_condition. Run B raises a hold, messages legal and leaves the order unscheduled: pass. Run C raises a hold but also schedules the order: fail on the ordering check.
**What it teaches:** pass criteria rest on final state and event order, so results are repeatable. Buyers can compare agents and prompts on the same snapshot.
State-based grading turns a vague idea of good behaviour into pass or fail checks that a buyer can repeat across runs.
- Audience:
- Licensed resellers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?