How should evaluation files separate agent observations from scoring information?
Separate observation files and scoring metadata help resellers prevent answer leakage and support reproducible evaluation across downstream customer implementations.
A generated evaluation release could separate `observations.jsonl` from restricted scoring material. Observation rows could contain `episode_id`, `visible_record_refs`, `task_prompt` and `policy_refs`. Scoring rows could contain `expected_action_set`, `forbidden_actions`, `required_evidence_refs` and `terminal_state_constraints`.
The expected result should not appear in filenames, record labels or conveniently named status fields visible to the agent. A reseller should also check whether answer text has been copied into retrieval metadata or marketplace previews. Hiding one answer column is insufficient if the same information remains accessible elsewhere.
An evaluation hook could capture `episode_id`, `action`, `arguments`, `observation_version` and `resulting_state_hash`. This supports inspection of what the agent knew and what its action changed. Where several actions are acceptable, scoring can validate membership in an allowed set rather than compare against one exact string.
MirrorFirm could provide these structures with a scorer interface and schema version. Downstream buyers can run the same inputs through different agents while preserving the distinction between accessible evidence and evaluator knowledge. Resellers should document which files belong in training, evaluation administration and agent-facing runtime storage.
Distribute a clear evaluation interface that preserves hidden criteria and records enough evidence to diagnose failures.
- Audience:
- Licensed resellers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should buyers run before using a generated task pack?
- Which evaluation hooks expose duplicate actions and unreliable recovery?
- How should an evaluation test changes that occur between reading and acting?
- How should follow-up quality be scored when several responses are acceptable?
- How can buyers tell whether an agent follows policy instead of superficial cues?
- What should we inspect before trusting an environment's evaluation results?