How can buyers evaluate correctness without requiring one exact action sequence?
Business invariants let buyers assess multiple valid action paths while still detecting broken relationships, unauthorised changes and unjustified completion of customer requests.
A customer change may have several valid execution paths. An agent might inspect the policy before opening a dependent record, or discover the same restriction through a validation action. Requiring one exact sequence can penalise correct behaviour without revealing a business failure.
Instead, define invariants over the environment. A closed request must have no mandatory propagation job awaiting action. A historical invoice must retain its issued recipient details unless the supplied correction process authorises replacement. A field requiring approval must not change before that approval exists.
Some invariants concern the final state; others must hold after every action. This distinction catches an agent that briefly removes a required restriction and later restores it. The final snapshot alone would miss the unsafe transition.
Buyers can inspect a generated pack by pairing each invariant with passing and failing traces. MirrorFirm could generate traces for this purpose, including different valid action orders. The buyer should confirm that the checks follow the stated business rules and that the evaluator accepts supported partial outcomes, rather than forcing every request into a completed state.
Score the conditions that must hold throughout the episode, allowing legitimate variation in how an agent reaches a valid outcome.
- Audience:
- Environment builders
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a scenario test whether an agent handles another worker's concurrent change?
- How can buyers check whether an agent understands changes rather than memorising wording?
- What should buyers inspect before trusting a generated change evaluation set?
- Which evaluation hooks expose retry and duplicate-processing failures?
- How should evaluation distinguish complete handling from a correct-looking final record?
- What should buyers inspect before trusting a generated evaluation pack?