Which quality checks should I run on a sample release before any agent runs?
A checklist of consistency, difficulty and grader checks to run on a sample set-up release, covering references, dates, near-duplicates, shortcuts and reset determinism.
Use this list on a sample release before you run any agent against it; each check targets a different way an environment can mislead you.
- Every reference field resolves to an existing record, with no orphaned accounts or obligations. - Dates are ordered correctly: approval before opening, expiry after start, due dates after creation. - Near-duplicate parties are close enough to cause a genuine matching error, not an obvious one. - Each policy rule can be followed to a single outcome by a person reading it cold. - A scripted correct trajectory passes the grader from a fresh reset. - Shortcut trajectories, such as activating before conditions are met, fail with a clear reason. - Unintended writes to unrelated records are detected and reported as their own category. - A second reset produces an identical starting state, including identifiers and timestamps.
Each check targets a different way an environment can mislead you, so failures are found before they contaminate agent scores.
- Audience:
- Environment builders
- Content type:
- Checklist
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- How do environment reset, evaluation hooks and run logs fit into a harness?
- What does state-based grading catch that reading the agent's reply would miss?