How do we wire set-up checks into an evaluation pipeline?

Evaluation needs a repeatable loop. This describes how a buyer, or a reseller testing a pack before listing it, can wire the checks into a pipeline.

**Load and validate.** Read the Parquet or JSONL tables, then run a schema check against the supplied schema file. A validator script can assert key resolution, enum values and timestamp order, and write failures to a report.

**Run the task.** For each task line, restore the snapshot named in `initial_state_ref`, give the agent the `prompt` and the `allowed_tools`, and capture every tool call and the resulting events.

**Score.** Compare final state with `expected_end_state` using named checks. A result record could look like:

- `run_id`, `task_id`, `agent_version` - `checks`: an array of `{name, passed, detail}` - `score`: a weighted total, with weights set in the pack - `policy_violations`: an array of rule references

**Aggregate.** Group results by exception type and by check name, so a regression in duplicate handling is visible even when the overall score is stable.

**Reproduce.** Store the snapshot identifier, schema version and pack version with each result, so a run can be replayed later.

Ask whether the scoring code is supplied or only the expected states, and under what terms your customers may modify it.