Skip to content
MirrorFirm

Which evaluation hooks make an interactive evidence task reproducible?

Private state assertions and structured action traces can assess how a model assembled its bundle, while reproducible resets let labs compare runs from equivalent starting conditions.

An evaluation harness can consume a submission containing `bundle_id`, `claim_findings`, `evidence_refs` and `unresolved_requests`. References should identify record versions and any cited spans so the evaluator can check what the model actually relied on.

Private hooks can inspect whether required claims have support, whether cited records were accessible during the run and whether the submitted bundle matches the final environment state. These assertions should remain outside tool responses visible to the model. Otherwise the evaluation machinery can disclose the answer it is meant to assess.

An action trace can record `action_id`, `tool_name`, `arguments`, `result_code` and `state_revision`. This supports diagnosis of behaviours such as submitting before a requested record arrives or citing material that the model never retrieved.

A reset contract can identify the scenario revision, initial state digest and event schedule used for each run. The lab can then replay comparable conditions and distinguish model variation from changing task state. Where several action sequences are valid, scoring should judge the resulting evidence position rather than require one exact sequence of tool calls.

Capture task outcomes and action-level errors without exposing the evaluator's answer state to the model.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Audit evidence assembly