Skip to content
MirrorFirm

What should a first delivery include so we can wire it into our harness?

Lists the files, manifest, tool definitions, snapshot, reference grader and responder script a first delivery needs, plus a sensible order for loading and testing.

A first delivery can be small and still be wired into your harness. Ask for these pieces so you can test the integration before volume matters.

- **Files**: the case, requirement, document, link, message and gold files as JSONL, or Parquet if you prefer, with a schema file listing types, nullability and enumerations - **Manifest**: release version, generation settings, a seed where reproducibility is supported, and a record count per file so truncation is detectable - **Tool definitions**: JSON schemas for the case actions your agent will call, matching the field names in the data - **Environment snapshot**: a starting state and a reset routine, so runs begin identically - **Reference grader**: a script that reads a trace and a gold file and reports decision, gap and request results separately - **Responder script**: scripted replies for multi-turn cases

Start by loading the files and running the validator against the schema. Then run one case end to end with a trivial agent, confirm the trace format and reset behave, and only then run the full sample.

Keep the data directory read-only for the agent process and the gold file out of its reach. If your harness uses different field names or a different status vocabulary, say so at the outset, so mappings are agreed in the release rather than patched afterwards.

Version each release and note the changes between versions, so results from different drops are not compared by accident.

A small first delivery that includes schemas, a reset routine and a reference grader lets you test the integration before volume matters.

Audience:
Evaluation teams
Content type:
Integration
Buyer question:
Scope and getting started

Content release: r-a220eb6e099b243b

More on Document sufficiency