Skip to content
MirrorFirm

How should reset and grader hooks work for schedule tasks?

Deterministic reset from a snapshot, a controllable clock and grader functions that read final state and the event log make schedule runs repeatable and comparable.

Three hooks make schedule tasks repeatable.

**Reset.** `reset(snapshot_id, seed)` loads the starting rows for a task and sets the random seed for anything stochastic, such as simulated customer replies. After a run, the environment should return to the exact snapshot, so test that two resets produce identical state hashes.

**Clock.** Expose `set_clock(timestamp)` and have all due-date, grace-period and cut-off logic read from it. Without a controllable clock, tasks that depend on 'today' cannot be replayed.

**Grader.** A function `grade(task_id, final_state, event_log)` returns a structured result such as `passed`, `score`, `invariant_failures`, `policy_violations` and `side_effects`. Keep each check a separate named function so failures can be counted by type.

Suggested checks include `sum_matches_principal`, `dates_ordered`, `links_resolve`, `approval_present_when_required` and `no_unrelated_changes`.

For training use, log each step as a JSONL line with the observation, the action, the new state reference and the reward components. For evaluation, hold back some tasks and graders so they are not reused in training.

The starting snapshots are generated, so run a validator over them at load time and fail early if an invariant is broken before the agent acts.

Repeatable resets and state-based graders let you compare models, prompts and tool designs on the same footing.

Audience:
Environment builders
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Repayment scheduling