How do evaluation hooks and environment resets work for set-up tasks?
A simulated set-up environment can expose tool actions, restore a seeded state on reset, log every call and run a grader that compares end state with expected records.
A simulated environment can be built around a small tool surface and a deterministic state store.
**Tool actions.** Typical calls are `search_party(query)`, `get_decision(decision_id)`, `create_party(fields)`, `create_account(party_id, decision_id, product_code)`, `set_status(account_id, status)`, `open_case(reason_code, refs)` and `send_handover(team, note)`. Each returns structured results, including realistic rejections with error codes.
**Reset.** `reset(task_id, seed)` restores the seeded starting state for that task. The same seed yields the same state, so a run can be repeated exactly.
**Logging.** Every call is written to a trace with a step number, arguments, response and the state version after the call. Traces export as JSONL.
**Grader hook.** After the run, `grade(task_id)` compares final state with `expected_state`. It returns:
- per-record match results - triggered forbidden actions - handover completeness - a failure-mode label where one applies
**Extra hooks.** A step budget, a read-only mode for inspecting state and a policy-version switch let you test sensitivity to the rule in force.
Agree the exact tool names, error codes and scoring outputs in a pilot, so they fit your harness.
Deterministic resets and call logs make runs repeatable and comparable across model versions, which is the basis of a usable evaluation.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?