What does a grader file and results schema look like for mismatch tasks?

Grading works best when the grader reads structured artefacts instead of free text. A release can ship a grader file per task alongside a model-facing task file.

The grader record can contain:

- `task_id` and `schema_version` - `target_state`: the table, `record_id` and field values that must hold at the end - `forbidden_changes`: records that must not change, including decoys - `required_links`: for instance an adjustment that must reference the original payment - `allowed_actions`: action names with argument constraints and the role authority - `expected_outcome` and `accepted_alternatives` - `root_cause` and `evidence_ids` for scoring the explanation - `side_effects`: downstream items that should be cancelled, created or left unchanged

A runner replays the agent's event log against the reset snapshot, then writes one result row per task with separate columns: `end_state_ok`, `no_forbidden_change`, `authority_ok`, `links_ok`, `cause_correct`, `evidence_overlap` (a score between 0 and 1) and `side_effects_ok`. Rows are written as JSONL or Parquet so they can be joined to your own run metadata, such as model version and sampling settings.

For free-text rationales, a rubric file can list the facts a good explanation must state, which a human reviewer or an LLM-based judge can check. Judge-based scores need their own calibration against human review.