How can an evaluation harness score review behaviour beyond the final answer?
Structured action traces and private scoring rules can assess evidence use, side effects and recovery, while keeping expected outcomes outside the agent's accessible business context.
An evaluation harness can capture events with `episode_id`, `action_id`, `actor_role`, `tool_name`, `input_refs`, `result_code` and `state_delta`. These fields let a scorer reconstruct what the agent inspected and changed without relying solely on its final prose.
Private evaluation data can specify acceptable terminal states, prohibited side effects and required evidence dependencies. Keep that data outside searchable business records. Otherwise, retrieval may expose the answer instead of the evidence needed to derive it.
For a scheduled account review, hooks can check whether a decision cites an accessible source, whether a required referral exists and whether an unrelated account changed. Some tasks can permit several action sequences, so scoring should focus on obligations and constraints rather than exact tool-call matching.
MirrorFirm could scope reset support around an immutable generated snapshot, a scenario seed and a fresh event log. Reset verification should confirm that decisions, messages and pending tasks from the previous run are absent. This gives labs repeatable episodes while preserving freedom for agents to choose different valid execution paths.
Score the operational path to a decision as well as the decision itself.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should precede training on generated review cases?
- How can a late event reveal whether an agent checks its assumptions before committing a decision?
- Which evaluation hooks make review runs reproducible and auditable?
- How should evaluation distinguish a sound review decision from a persuasive explanation?
- How can we tell whether an agent responds to meaningful evidence rather than superficial cues?
- How should buyers inspect realism and disclosure risk in generated review material?