How would a pilot environment be delivered, reset and connected to our harness?
A pilot can ship data files, a schema description, a grader and an optional tool-based environment with seeded reset, so runs are repeatable inside a lab's existing harness.
Agree these integration points before the pilot starts.
**Delivery**
- Data as Parquet tables or nested JSONL, with a schema description and a closed list of reason codes. - A `split` field marking train, dev and test cases. - Gold labels in a separate file keyed by `case_id`.
**Tool actions for an environment (optional)**
- `get_case(case_id)` returns the case record and document list. - `read_document(doc_id)` returns content. - `lookup_requirement(policy_id, case_type)` returns the requirement list. - `send_request(case_id, recipient_role, items[])` returns a simulated reply. - `set_status(case_id, status)` records the decision outcome. - `escalate(case_id, reason, to_role)` records an escalation.
**Reset and repeatability**
- Each episode starts from a `seed` that restores case status, message history and reply behaviour. - A reset call returns the environment to the initial state, and a log records every tool call for replay.
**Evaluation hooks**
- The grader reads the final case state and the tool-call log, and writes JSONL with `run_id`, `model_id`, `case_id` and per-case measures. - A turn limit and a malformed-output rule are set in configuration.
Confirm file formats, tool names, the harness language and how results are shared in the pilot scope. These details should follow your existing setup, and nothing here fixes terms or timing.
Clear delivery and reset mechanics let engineers run the pilot quickly and repeat any result exactly.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- How do environment reset, seeds and versioning work for a sufficiency environment?
- What licence terms and limits should I expect for a sufficiency release?
- How should I scope a sufficiency dataset, task pack or environment?
- What does a useful first pilot look like for a team building a utility onboarding agent?
- What should I ask for in a sufficiency sample or pilot?
- What should a first handoff package contain and how do we integrate it?