Skip to content
MirrorFirm

How can I inspect the quality of a set-up environment before using it?

Covers three places to inspect quality, namely the starting state, the policy rules and the grader, plus the failure modes and the grounding review a buyer should carry out.

Quality in a set-up environment shows up in three places: the starting state, the rules and the grading.

**Starting state.** Check that records are internally consistent. Every reference should resolve, dates should be ordered sensibly, and statuses should match the documents behind them. Then check that the seeded difficulty is real: near-duplicate parties should be close enough to cause a mistake, not obviously different.

**Rules.** Read the policies as a new operator would. Can a person follow them to a single correct outcome, or are two readings plausible? Ambiguous rules produce noisy grades.

**Grading.** Run a few tasks by hand, or with a simple scripted baseline, and confirm the grader accepts the correct path and rejects shortcuts such as activating before a condition is met. Check that it also accepts legitimate alternatives, such as a different but valid order of actions.

Then look at failure modes that matter for training:

- success reported in text without the state change - duplicate creation after a retry - changes to records the task did not mention - conditions in free-text notes ignored

Finally, inspect grounding and disclosure risk. Because the material is generated, ask how the structure and exceptions were informed, and review samples for anything that looks like it could identify a real party. Generation does not remove the need for that review.

Check consistency, rule clarity and grader behaviour on a known-correct path before trusting any score an agent earns.

Audience:
Environment builders
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Account setup