How should I score an agent on conditional approvals beyond a single pass rate?

Evaluating an agent on conditional approvals is mainly about two questions: does it proceed when it should, and does it hold when it should not?

A single pass rate hides this. An agent that always waits scores well on blocked cases and poorly on cleared ones, and an agent that always proceeds does the reverse. Report the two separately, then look at the cases between them.

Useful measures include:

- **Premature action:** a gated step taken while a condition is open or unresolved. - **Unnecessary hold:** a step withheld after every condition was properly met. - **Correct reason:** whether the stated reason for holding names the right condition. - **Escalation fit:** whether the agent routes a referred or unmeetable condition to the right role. - **State fidelity:** whether the records the agent writes leave the condition and evidence states accurate.

Because generated tasks carry ground truth, each of these can be scored automatically, with human review reserved for ambiguous cases.

Also inspect the data itself. Check that conditions and evidence are internally consistent, that dates make sense, that policies don't contradict one another, and that the difficulty is spread across easy and hard cases. Generated material can be too tidy, so ask whether the exceptions look like those your teams actually handle, and treat any gap as a scoping issue for the next release.