Skip to content
MirrorFirm

How do reset, tool actions and graders work in a simulated allocation environment?

A simulated environment needs deterministic reset, a defined tool set, structured observations, state-based graders and per-step logs so scores can be compared across runs and policy variants.

A simulated environment for this workflow needs predictable state, so scores can be compared across runs.

**Reset.** `reset(case_id, seed)` restores receipts, open items, policy version and case notes to the starting state. The same seed should yield the same state, which a buyer can confirm by hashing a snapshot.

**Actions.** A typical tool set: `list_open_items(account_id)`, `get_policy(rule_id)`, `apply_payment(receipt_id, item_id, amount)`, `split_receipt(receipt_id, parts)`, `place_in_suspense(receipt_id, amount, reason)`, `raise_query(account_id, text)`, `request_approval(action_id, approver_role)` and `reverse_allocation(allocation_id)`.

**Observations.** Each call returns updated balances, a unique `event_id` and any blocking rule. Keep errors structured, such as `error=NEEDS_APPROVAL, rule=POL-DISP-02`, so a grader can see how the model reacts.

**Graders.** Run after the episode and read the final state, not the transcript. Typical checks are reconciliation, allocation match against the accepted set, authority respected, and downstream flags consistent.

**Logging.** Write one JSONL line per step with `episode_id`, `step`, `action`, `args`, `result` and `state_hash`. This supports replay and trajectory-level analysis.

**Variants.** Hold the data fixed and change the policy version, or the reverse, to separate reading skill from memorised habit.

Run a few episodes by hand first to check that the graders accept every valid path, not only the one the author expected.

Deterministic reset and state-based graders make runs comparable and let you vary policy or data independently.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Payment allocation