How should a model lab scope a first engagement and what limits apply?
Scoping begins with the behaviour you want to test, then selects format, domain and difficulty. This entry explains the limits of generated output and a sensible first step.
Scoping starts with a narrow question about what you want the model to do, then works back to the material needed.
Decide first whether you need a static dataset, a task and evaluation pack, a simulated environment, or a combination. Training material usually needs breadth of cases; an evaluation pack needs verified labels and graders; an environment needs tools, state and reset.
Then choose the domain or domains, the conditions and policies in play, and the mix of difficult cases. Starting with one domain and a defined set of failure modes makes the first review easier than trying to cover everything.
Limits apply. Output is generated, so it reflects a model of how businesses operate and not a record of any real case. It should not be presented as real business records, and it does not by itself establish compliance with any rule. Licence terms, delivery format, volume and schedule are agreed in a scoping conversation; this page does not set them.
A useful first step is a small sample or pilot: a handful of cases in your preferred format, with the policy text and labels, which your team reads and runs against a current model. The result tells you whether to widen, narrow or change the request.
Start with one domain, a defined set of failure modes and a small reviewed sample, then widen or change scope based on what you learn.
- Audience:
- Model labs
- Content type:
- Explanation
- Buyer question:
- Scope and getting started
Content release: r-a220eb6e099b243b
Related questions
- What does a first pilot review look like for a referral-processing agent team?
- How is generated conditional-approval material licensed, and what limits apply?
- What should a pilot request include so the first sample can be judged quickly?
- How should I scope a first pilot for conditional-approval material?
- How do I wire a pilot sample into my agent harness for a first run?
- What should I ask for in a first sample or pilot?