How do reset, seeding and grader hooks work for repeatable episodes?
Each episode starts from a snapshot loaded by task identifier, runs under a seed, and ends with a grader hook that diffs state against a reference and emits failure codes.
Repeatable episodes need three things: a reset, a seed and a grader hook.
**Reset**
Each task references a starting snapshot: the rows from `accounts`, `open_items`, `payments`, `cases` and `policies` that apply. `reset(task_id)` truncates the working tables and reloads that snapshot. Audit rows are cleared too, otherwise the next episode inherits earlier writes. A post-reset check compares row counts and a hash of the table contents against the snapshot manifest.
**Seed**
Where an environment includes any variable behaviour, such as the order in which search results are returned or simulated payer replies to queries, a `seed` field on the task fixes it. Record the seed with each run.
**Grader hook**
`grade(task_id)` runs when the episode ends:
1. Reads final state from the working tables. 2. Loads the `reference_outcomes` entry, including accepted alternatives. 3. Diffs allocations, statuses, unapplied balance and side records. 4. Emits a result object with `pass`, `score`, `failure_codes` and `process_metrics` such as rejected calls and reversals.
Store results as JSONL, one line per episode, with task id, seed, agent version and result, so runs can be compared. These interfaces are a proposal and can be adapted to your harness in a pilot. Generated records still need your own checks for grounding and disclosure risk.
A snapshot-based reset and a state-diff hook make runs repeatable and comparable across agent versions and training changes.
- Audience:
- Environment builders
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do I judge whether generated remittance and allocation records are realistic enough?
- Which failure modes should I test for in a payment matching agent?
- How would an evaluation score an agent on a hospital insurer remittance?
- What reset, logging and grading hooks should an evaluation environment expose?
- How should scoring handle cases where several allocations are defensible?
- How do I check generated data for grounding and disclosure risk?