Skip to content
MirrorFirm

How can evaluation recognise different valid ways to complete a correction?

Evaluation can combine business invariants with permitted transition rules, allowing different valid action sequences without accepting an unjustified final balance.

A useful evaluator separates properties that must always hold from choices the agent is allowed to make. An invariant might require that an adjustment references an existing source line and that its lines balance within the configured accounting rules. A transition rule might prohibit posting before the required decision exists.

The evaluator should also define terminal outcomes. Completion, justified referral and a documented inability to proceed can each be valid in different cases. Treating every unposted adjustment as failure would reward agents for acting despite missing prerequisites.

Some tasks permit more than one action order. An agent may retrieve policy before inspecting supporting evidence, or reverse that order, without changing the validity of its work. Scoring only against one reference transcript can penalise these legitimate paths.

A buyer can inspect an acceptance specification that names required final-state properties, prohibited transitions and unresolved obligations. Human review can then focus on explanations that the automated checks cannot settle. This produces a clearer distinction between business errors, harmless procedural variation and weaknesses in the evaluator itself.

Assess whether the resulting state and the route taken satisfy the task's business requirements.

Audience:
Model labs
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Ledger adjustment