Skip to content
MirrorFirm

How should evaluation files separate agent observations from scoring information?

Separate observation files and scoring metadata help resellers prevent answer leakage and support reproducible evaluation across downstream customer implementations.

A generated evaluation release could separate `observations.jsonl` from restricted scoring material. Observation rows could contain `episode_id`, `visible_record_refs`, `task_prompt` and `policy_refs`. Scoring rows could contain `expected_action_set`, `forbidden_actions`, `required_evidence_refs` and `terminal_state_constraints`.

The expected result should not appear in filenames, record labels or conveniently named status fields visible to the agent. A reseller should also check whether answer text has been copied into retrieval metadata or marketplace previews. Hiding one answer column is insufficient if the same information remains accessible elsewhere.

An evaluation hook could capture `episode_id`, `action`, `arguments`, `observation_version` and `resulting_state_hash`. This supports inspection of what the agent knew and what its action changed. Where several actions are acceptable, scoring can validate membership in an allowed set rather than compare against one exact string.

MirrorFirm could provide these structures with a scorer interface and schema version. Downstream buyers can run the same inputs through different agents while preserving the distinction between accessible evidence and evaluator knowledge. Resellers should document which files belong in training, evaluation administration and agent-facing runtime storage.

Distribute a clear evaluation interface that preserves hidden criteria and records enough evidence to diagnose failures.

Audience:
Licensed resellers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Overdue follow up