How do reset, tool actions and graders work in a simulated allocation environment?
A simulated environment needs deterministic reset, a defined tool set, structured observations, state-based graders and per-step logs so scores can be compared across runs and policy variants.
A simulated environment for this workflow needs predictable state, so scores can be compared across runs.
**Reset.** `reset(case_id, seed)` restores receipts, open items, policy version and case notes to the starting state. The same seed should yield the same state, which a buyer can confirm by hashing a snapshot.
**Actions.** A typical tool set: `list_open_items(account_id)`, `get_policy(rule_id)`, `apply_payment(receipt_id, item_id, amount)`, `split_receipt(receipt_id, parts)`, `place_in_suspense(receipt_id, amount, reason)`, `raise_query(account_id, text)`, `request_approval(action_id, approver_role)` and `reverse_allocation(allocation_id)`.
**Observations.** Each call returns updated balances, a unique `event_id` and any blocking rule. Keep errors structured, such as `error=NEEDS_APPROVAL, rule=POL-DISP-02`, so a grader can see how the model reacts.
**Graders.** Run after the episode and read the final state, not the transcript. Typical checks are reconciliation, allocation match against the accepted set, authority respected, and downstream flags consistent.
**Logging.** Write one JSONL line per step with `episode_id`, `step`, `action`, `args`, `result` and `state_hash`. This supports replay and trajectory-level analysis.
**Variants.** Hold the data fixed and change the policy version, or the reverse, to separate reading skill from memorised habit.
Run a few episodes by hand first to check that the graders accept every valid path, not only the one the author expected.
Deterministic reset and state-based graders make runs comparable and let you vary policy or data independently.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How do I judge whether generated remittance and allocation records are realistic enough?
- Which failure modes should I test for in a payment matching agent?
- How would an evaluation score an agent on a hospital insurer remittance?
- What reset, logging and grading hooks should an evaluation environment expose?
- How should scoring handle cases where several allocations are defensible?
- How do I check generated data for grounding and disclosure risk?