Skip to content
MirrorFirm

How do we evaluate whether an agent changed the right work?

Evaluation should distinguish missed affected work from unnecessary changes, because broad updates can appear complete while damaging valid decisions and unaffected cases.

Start with a reference set that classifies work as requiring change, requiring review or unaffected. Those categories should follow the release's policy semantics, with evidence for each classification. The agent's final state can then be compared with the required treatment rather than simply counting updates.

Coverage asks whether all work requiring action reached an acceptable state. Precision asks whether the agent avoided changing records that should have remained as they were. Both matter: an agent that reopens every completed decision may catch affected work while causing unnecessary disruption.

Review-required cases need separate treatment. If the evidence does not support an automatic decision, a documented referral may be the correct outcome. Forcing those cases into changed or unchanged categories would encourage unjustified certainty.

MirrorFirm could generate evaluation annotations and explanations kept outside the agent's observation surface. Builders can inspect disagreements at record level and trace them back to the applicable clause. This produces useful failure analysis: a missed dependency, a misunderstood transition rule and an excessive update can receive different diagnoses even when each leaves the episode incomplete.

Reward selective, justified propagation rather than the volume of records an agent modifies.

Audience:
Environment builders
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Policy change propagation