Skip to content
MirrorFirm

How would a pilot environment be delivered, reset and connected to our harness?

A pilot can ship data files, a schema description, a grader and an optional tool-based environment with seeded reset, so runs are repeatable inside a lab's existing harness.

Agree these integration points before the pilot starts.

**Delivery**

- Data as Parquet tables or nested JSONL, with a schema description and a closed list of reason codes. - A `split` field marking train, dev and test cases. - Gold labels in a separate file keyed by `case_id`.

**Tool actions for an environment (optional)**

- `get_case(case_id)` returns the case record and document list. - `read_document(doc_id)` returns content. - `lookup_requirement(policy_id, case_type)` returns the requirement list. - `send_request(case_id, recipient_role, items[])` returns a simulated reply. - `set_status(case_id, status)` records the decision outcome. - `escalate(case_id, reason, to_role)` records an escalation.

**Reset and repeatability**

- Each episode starts from a `seed` that restores case status, message history and reply behaviour. - A reset call returns the environment to the initial state, and a log records every tool call for replay.

**Evaluation hooks**

- The grader reads the final case state and the tool-call log, and writes JSONL with `run_id`, `model_id`, `case_id` and per-case measures. - A turn limit and a malformed-output rule are set in configuration.

Confirm file formats, tool names, the harness language and how results are shared in the pilot scope. These details should follow your existing setup, and nothing here fixes terms or timing.

Clear delivery and reset mechanics let engineers run the pilot quickly and repeat any result exactly.

Audience:
Model labs
Content type:
Integration
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Document sufficiency