Skip to content
MirrorFirm

How can an evaluation harness score review behaviour beyond the final answer?

Structured action traces and private scoring rules can assess evidence use, side effects and recovery, while keeping expected outcomes outside the agent's accessible business context.

An evaluation harness can capture events with `episode_id`, `action_id`, `actor_role`, `tool_name`, `input_refs`, `result_code` and `state_delta`. These fields let a scorer reconstruct what the agent inspected and changed without relying solely on its final prose.

Private evaluation data can specify acceptable terminal states, prohibited side effects and required evidence dependencies. Keep that data outside searchable business records. Otherwise, retrieval may expose the answer instead of the evidence needed to derive it.

For a scheduled account review, hooks can check whether a decision cites an accessible source, whether a required referral exists and whether an unrelated account changed. Some tasks can permit several action sequences, so scoring should focus on obligations and constraints rather than exact tool-call matching.

MirrorFirm could scope reset support around an immutable generated snapshot, a scenario seed and a fresh event log. Reset verification should confirm that decisions, messages and pending tasks from the previous run are absent. This gives labs repeatable episodes while preserving freedom for agents to choose different valid execution paths.

Score the operational path to a decision as well as the decision itself.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Periodic review