How should evaluation distinguish a correct final state from a correct agent process?
An agent can reach an acceptable final state through unauthorised or unsupported actions, so evaluation needs separate outcome and process criteria.
A final record can look correct even when the agent used an unacceptable route to produce it. An agent might remove a review requirement, edit a protected field directly and then restore the expected status. A snapshot-only check would miss the bypass.
Evaluation should therefore separate outcome correctness from action validity. Outcome criteria ask whether affected work now follows the applicable policy. Process criteria ask whether the agent had the required authority, used supported tools and respected mandatory review steps along the way.
For a financial-services task, the expected result may be a revised assessment awaiting approval. An agent that grants the approval itself has exceeded its role, even if its assessment is otherwise sound. The evaluation should preserve that distinction instead of rewarding a more advanced status automatically.
A generated task pack can provide allowed-action rules, protected transitions and evaluator-visible action traces. Developers can inspect failures by category and decide whether they arise from policy interpretation, permission handling or execution. This produces more useful feedback than a single label that treats every unsuccessful run as the same kind of mistake.
Score the result and the action path separately to expose agents that succeed only by bypassing operational controls.
- Audience:
- Agent developers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a buyer inspect business grounding and disclosure risk in generated material?
- What evaluation hooks help test recovery from partial policy-update failures?
- How can a policy-change scenario test whether an agent respects concurrent edits?
- Which checks reveal whether an agent understands applicability rather than memorising cues?
- How do we evaluate whether an agent changed the right work?
- Which paired tests reveal whether an agent understands policy applicability?