What grading inputs and result records support evaluation of a schedule task pack?

Evaluation works best when the grader reads the same files your customers load. A task pack can ship a small set of structured grading inputs alongside the data.

A task definition in JSONL might carry:

- `task_id`, `instruction`, `start_snapshot` - `allowed_actions`, a list of tool names - `expected_final_state`, a reference to the target schedule version - `required_steps`, such as approval before commit - `forbidden_effects`, such as a notice sent for an unapproved plan - `tolerance`, the rounding allowance in currency units

A grader can then produce a result record per run: `run_id`, `task_id`, `final_schedule_match`, `step_order_ok`, `policy_violations`, `rounding_delta` and `notes`. Storing these as Parquet makes comparison across model versions straightforward.

Useful metrics to ask for:

- exact match of the final instalment lines - match on due dates only, to separate date logic from arithmetic - policy adherence across tasks that should be refused - recovery after a tool error, using a fault-injected step

Request the grader code or specification in the pilot, plus a reset method for each task. A reseller can then offer evaluation-ready packs, with clear notes that the underlying content is generated and that grading logic should be reviewed by the buyer.