Skip to content
MirrorFirm

How do evaluation hooks and environment resets work for set-up tasks?

A simulated set-up environment can expose tool actions, restore a seeded state on reset, log every call and run a grader that compares end state with expected records.

A simulated environment can be built around a small tool surface and a deterministic state store.

**Tool actions.** Typical calls are `search_party(query)`, `get_decision(decision_id)`, `create_party(fields)`, `create_account(party_id, decision_id, product_code)`, `set_status(account_id, status)`, `open_case(reason_code, refs)` and `send_handover(team, note)`. Each returns structured results, including realistic rejections with error codes.

**Reset.** `reset(task_id, seed)` restores the seeded starting state for that task. The same seed yields the same state, so a run can be repeated exactly.

**Logging.** Every call is written to a trace with a step number, arguments, response and the state version after the call. Traces export as JSONL.

**Grader hook.** After the run, `grade(task_id)` compares final state with `expected_state`. It returns:

- per-record match results - triggered forbidden actions - handover completeness - a failure-mode label where one applies

**Extra hooks.** A step budget, a read-only mode for inspecting state and a policy-version switch let you test sensitivity to the rule in force.

Agree the exact tool names, error codes and scoring outputs in a pilot, so they fit your harness.

Deterministic resets and call logs make runs repeatable and comparable across model versions, which is the basis of a usable evaluation.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Account setup