How can developers evaluate a correction beyond checking the final balance?
Evaluation should inspect business invariants and the permitted set of changes, because a correct final balance can conceal damage elsewhere in the records.
A final balance can be correct for the wrong reasons. An agent might remove an unrelated entry, alter a source amount or create offsetting errors that happen to cancel. Evaluation needs to inspect the allowed change set as well as the requested outcome.
For a financial posting task, the supplied accounting model can require balanced journal components and unchanged source documents. For an operational quantity correction, the invariant might require total stock across locations to remain constant. These are task-specific rules, not assumptions to apply indiscriminately across domains.
A generated evaluation pack could define protected records, permitted fields and expected relationships between the opening and closing state. It can compare the full state difference against those constraints. A successful run would resolve the target discrepancy without modifying unrelated customer accounts, allocation dimensions or historical evidence.
MirrorFirm could include evaluator explanations showing which invariant each check protects. Buyers can inspect whether the checks match the intended business process and whether legitimate alternative solutions are accepted. This approach is especially useful for agents with broad tool access: it measures the boundaries of their actions, not just whether they eventually produce the requested number.
Catch compensating errors and unintended mutations that an end balance check would miss.
- Audience:
- Agent developers
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What should buyers inspect before trusting a generated correction pack?
- How should an agent behave when a posting request times out?
- How do we distinguish a justified pause from an agent avoiding the task?
- How should evaluation hooks avoid leaking the expected correction to the agent?
- Which hooks reveal duplicate posting after a tool timeout?
- How can an evaluation expose stale decisions during concurrent corrections?