How can evaluators recognise valid corrections without requiring one exact action sequence?
Outcome constraints and preserved invariants allow evaluators to recognise multiple valid correction paths while detecting collateral changes hidden by a correct closing balance.
Start with the properties that must hold after a valid correction. These can include the intended account allocation, preservation of the original source entry and a complete link between the correction and its supporting decision. Then specify what must remain unchanged, such as unrelated customer balances or supplier payment instructions.
This supports evaluation of more than one valid execution path. An agent might gather supporting records in a different order or prepare a draft before requesting approval. Those differences need not count as failures if the resulting state and required controls are correct.
Keep forbidden side effects separate from the main outcome. An agent that fixes the target balance while modifying an unrelated account should fail that constraint even if its target arithmetic is exact. A single aggregate score can otherwise hide a serious boundary error.
Generated cases can include evaluator-only invariants and a state-difference report. Reviewers can inspect which properties passed, which failed and what changed unexpectedly. This approach also makes disputed scores easier to diagnose because the judgement points to a business requirement rather than an arbitrary reference sequence.
Score the meaning and boundaries of the final state rather than matching a single reference trace.
- Audience:
- Evaluation teams
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can developers evaluate a correction beyond checking the final balance?
- What should buyers inspect before trusting a generated correction pack?
- How should an agent behave when a posting request times out?
- How do we distinguish a justified pause from an agent avoiding the task?
- How should evaluation hooks avoid leaking the expected correction to the agent?
- Which hooks reveal duplicate posting after a tool timeout?