What should a first delivery include so we can wire it into our harness?
Lists the files, manifest, tool definitions, snapshot, reference grader and responder script a first delivery needs, plus a sensible order for loading and testing.
A first delivery can be small and still be wired into your harness. Ask for these pieces so you can test the integration before volume matters.
- **Files**: the case, requirement, document, link, message and gold files as JSONL, or Parquet if you prefer, with a schema file listing types, nullability and enumerations - **Manifest**: release version, generation settings, a seed where reproducibility is supported, and a record count per file so truncation is detectable - **Tool definitions**: JSON schemas for the case actions your agent will call, matching the field names in the data - **Environment snapshot**: a starting state and a reset routine, so runs begin identically - **Reference grader**: a script that reads a trace and a gold file and reports decision, gap and request results separately - **Responder script**: scripted replies for multi-turn cases
Start by loading the files and running the validator against the schema. Then run one case end to end with a trivial agent, confirm the trace format and reset behave, and only then run the full sample.
Keep the data directory read-only for the agent process and the gold file out of its reach. If your harness uses different field names or a different status vocabulary, say so at the outset, so mappings are agreed in the release rather than patched afterwards.
Version each release and note the changes between versions, so results from different drops are not compared by accident.
A small first delivery that includes schemas, a reset routine and a reference grader lets you test the integration before volume matters.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- How do environment reset, seeds and versioning work for a sufficiency environment?
- What licence terms and limits should I expect for a sufficiency release?
- How should I scope a sufficiency dataset, task pack or environment?
- What does a useful first pilot look like for a team building a utility onboarding agent?
- What should I ask for in a sufficiency sample or pilot?
- What should a first handoff package contain and how do we integrate it?