How should success be measured when several correction paths are valid?
Evaluation can accept multiple justified correction paths while requiring each successful run to satisfy authority, accounting and audit obligations throughout execution.
A reference action sequence is useful for debugging, but it should not automatically become the only accepted solution. A correction may be valid through either a reversal and replacement or a permitted compensating entry. The relevant question is whether the chosen route satisfies the scoped business rules.
Define success as a set of obligations. These can include restoring the intended accounting position, obtaining the required authority and preserving an understandable link to the original entry. Some obligations concern the final state; others must hold throughout the run.
Keep hard violations separate from efficiency measures. An unauthorised intermediate posting should not disappear from the assessment because the agent later reverses it. Conversely, an extra evidence read need not invalidate an otherwise sound correction.
A buyer can ask for an evaluator that recognises alternative valid paths and explains which obligation failed. Inspect disputed runs manually against the policy pack. This helps distinguish an agent failure from an evaluator that mistakes its preferred procedure for the only legitimate procedure.
Score valid business outcomes without forcing every agent to imitate one reference sequence.
- Audience:
- Environment builders
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can developers evaluate a correction beyond checking the final balance?
- What should buyers inspect before trusting a generated correction pack?
- How should an agent behave when a posting request times out?
- How do we distinguish a justified pause from an agent avoiding the task?
- How should evaluation hooks avoid leaking the expected correction to the agent?
- Which hooks reveal duplicate posting after a tool timeout?