How do we evaluate whether an agent changed the right work?
Evaluation should distinguish missed affected work from unnecessary changes, because broad updates can appear complete while damaging valid decisions and unaffected cases.
Start with a reference set that classifies work as requiring change, requiring review or unaffected. Those categories should follow the release's policy semantics, with evidence for each classification. The agent's final state can then be compared with the required treatment rather than simply counting updates.
Coverage asks whether all work requiring action reached an acceptable state. Precision asks whether the agent avoided changing records that should have remained as they were. Both matter: an agent that reopens every completed decision may catch affected work while causing unnecessary disruption.
Review-required cases need separate treatment. If the evidence does not support an automatic decision, a documented referral may be the correct outcome. Forcing those cases into changed or unchanged categories would encourage unjustified certainty.
MirrorFirm could generate evaluation annotations and explanations kept outside the agent's observation surface. Builders can inspect disagreements at record level and trace them back to the applicable clause. This produces useful failure analysis: a missed dependency, a misunderstood transition rule and an excessive update can receive different diagnoses even when each leaves the episode incomplete.
Reward selective, justified propagation rather than the volume of records an agent modifies.
- Audience:
- Environment builders
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a buyer inspect business grounding and disclosure risk in generated material?
- What evaluation hooks help test recovery from partial policy-update failures?
- How should evaluation distinguish a correct final state from a correct agent process?
- How can a policy-change scenario test whether an agent respects concurrent edits?
- Which checks reveal whether an agent understands applicability rather than memorising cues?
- Which paired tests reveal whether an agent understands policy applicability?