Skip to content
MirrorFirm

What elements should a lab request in a set-up task pack?

A checklist of elements a buyer can request when specifying a set-up task pack, from starting state and policies to graders and difficulty variation.

A task pack is only as useful as the state and rules behind it. When specifying one, a buyer can ask for the following:

- **Seeded starting state:** parties, decisions, accounts and open cases loaded before each task begins. - **Policy documents:** versioned rules referenced by identifier from each task. - **Tool definitions:** the actions the model may call, with argument schemas and error responses. - **Exception mix:** a stated spread of clean cases, near-duplicates, unmet conditions and rejections. - **Expected end state:** the records, statuses and links a correct run leaves behind. - **Grader logic:** state comparison rules, plus checks for forbidden actions such as early activation. - **Difficulty variation:** the same scenario with more or fewer conflicting records, so results can be compared. - **Split labels:** a train, dev and test marking, with held-out policy variants.

A pack can also carry domain variation, such as lending, utilities or professional services, if you want to test whether a model transfers the skill. Everything is generated, so ask how each element was grounded and plan a disclosure review of the sample before scaling up.

Naming these elements up front turns a vague request for realistic data into a specification a pilot can be built and judged against.

Audience:
Model labs
Content type:
Checklist
Buyer question:
What can be generated

Content release: r-a220eb6e099b243b

More on Account setup