How do I score an agent's schedule edit fairly?

**How do I score an agent's schedule edit fairly?**

Use layers, and report them separately so you can see where a model fails.

- **Final-state match.** Compare the resulting records with the expected end state, field by field. Where several end states are valid, define the allowed set or test invariants instead of one exact snapshot. - **Invariants.** Totals reconcile, dates are ordered, links resolve, balances agree. - **Policy compliance.** Check that the action was permitted, that approvals were requested when a threshold applied, and that the correct policy version was used. - **Escalation correctness.** For tasks where the right answer is to refuse or hand over, score whether the agent did so and gave a usable reason. - **Side effects.** Penalise unnecessary changes, such as touching unrelated plans or sending extra notices. - **Process.** Number of tool calls, wasted reads and retries can be tracked, but keep them out of the correctness score.

Partial credit is sensible where tasks have several steps, as long as each step has its own check.

If a task cannot be scored without human judgement, flag it as such in the pack. It is better to say so than to hide ambiguity in the grader.