How should I score an agent that clears some conditions but not all?
Score the reasoning path and final state separately, give partial credit for correct intermediate steps, and treat any gated action executed too early as a hard failure.
**Should partial progress earn credit?**
Yes, but keep it separate from the outcome. Give credit for each correct intermediate step, such as identifying the open condition, requesting the right evidence, or routing a waiver to the right role.
**What counts as a hard failure?**
Executing a gated action while any blocking condition is open. Also clearing a condition without authority, or accepting evidence that fails the evidence spec. These should cap the task score regardless of other steps.
**What about a correct refusal?**
When the right answer is that work cannot proceed, a clear explanation that names the blocking condition should score as a full success. Otherwise you train agents to push forward.
**How do I handle several valid paths?**
Store acceptable orderings in the expected-trace field, and score on constraints such as gate checked before action rather than one exact sequence.
**What should I report?**
Per-task path score, final-state match, count of hard failures by type, and the share of tasks where the agent stated its reasoning correctly. Review a sample of transcripts by hand, because automated checks can miss a wrong explanation attached to a right action.
Separating path scores from outcome scores shows whether an agent was careful or merely lucky, which a single pass or fail cannot.
- Audience:
- Environment builders
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What trace format and grader hooks make conditional-approval evaluation automatic?
- How do I check that generated tasks are hard enough and do not leak the answer?
- How should I score an agent on conditional approvals beyond a single pass rate?
- Which checks should I run on a sample before committing to wider use?
- What does a failed evaluation run look like when an agent binds a policy too early?
- How do I wire graders and reset into a conditional-approval environment?