How can I inspect the quality of a set-up environment before using it?
Covers three places to inspect quality, namely the starting state, the policy rules and the grader, plus the failure modes and the grounding review a buyer should carry out.
Quality in a set-up environment shows up in three places: the starting state, the rules and the grading.
**Starting state.** Check that records are internally consistent. Every reference should resolve, dates should be ordered sensibly, and statuses should match the documents behind them. Then check that the seeded difficulty is real: near-duplicate parties should be close enough to cause a mistake, not obviously different.
**Rules.** Read the policies as a new operator would. Can a person follow them to a single correct outcome, or are two readings plausible? Ambiguous rules produce noisy grades.
**Grading.** Run a few tasks by hand, or with a simple scripted baseline, and confirm the grader accepts the correct path and rejects shortcuts such as activating before a condition is met. Check that it also accepts legitimate alternatives, such as a different but valid order of actions.
Then look at failure modes that matter for training:
- success reported in text without the state change - duplicate creation after a retry - changes to records the task did not mention - conditions in free-text notes ignored
Finally, inspect grounding and disclosure risk. Because the material is generated, ask how the structure and exceptions were informed, and review samples for anything that looks like it could identify a real party. Generation does not remove the need for that review.
Check consistency, rule clarity and grader behaviour on a known-correct path before trusting any score an agent earns.
- Audience:
- Environment builders
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?