How should reset and grader hooks work for schedule tasks?
Deterministic reset from a snapshot, a controllable clock and grader functions that read final state and the event log make schedule runs repeatable and comparable.
Three hooks make schedule tasks repeatable.
**Reset.** `reset(snapshot_id, seed)` loads the starting rows for a task and sets the random seed for anything stochastic, such as simulated customer replies. After a run, the environment should return to the exact snapshot, so test that two resets produce identical state hashes.
**Clock.** Expose `set_clock(timestamp)` and have all due-date, grace-period and cut-off logic read from it. Without a controllable clock, tasks that depend on 'today' cannot be replayed.
**Grader.** A function `grade(task_id, final_state, event_log)` returns a structured result such as `passed`, `score`, `invariant_failures`, `policy_violations` and `side_effects`. Keep each check a separate named function so failures can be counted by type.
Suggested checks include `sum_matches_principal`, `dates_ordered`, `links_resolve`, `approval_present_when_required` and `no_unrelated_changes`.
For training use, log each step as a JSONL line with the observation, the action, the new state reference and the reward components. For evaluation, hold back some tasks and graders so they are not reused in training.
The starting snapshots are generated, so run a validator over them at load time and fail early if an invariant is broken before the agent acts.
Repeatable resets and state-based graders let you compare models, prompts and tool designs on the same footing.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?