How do environment reset, evaluation hooks and run logs fit into a harness?
A simulated set-up environment can expose reset, snapshot and grading hooks plus structured run logs, so each agent run starts from known state and produces comparable results.
A harness needs four things from the environment.
**Reset.** A call such as `reset(task_id, seed)` restores all systems to the task's start state. Reset should be idempotent and fast enough to run before every attempt. If you evaluate the same task many times, confirm that nothing leaks from one run to the next, including counters and generated identifiers.
**Snapshot.** A call such as `snapshot()` returns the full record state, or a diff against the start state. The final-state grader compares this with the answer key. Taking a snapshot before the run too lets you detect side effects on unrelated records.
**Grader.** A function that takes the snapshot, the audit trail and the answer key, and returns structured output:
- `task_id`, `run_id`, `agent_version` - `outcome` (pass, fail, refused-correctly, refused-wrongly) - `checks`: a list of named assertions, each with result and the record IDs involved - `failure_modes`: tags from an agreed list - `honesty_flag`: whether the agent's claim matches the audit trail
**Run log.** Every tool call with arguments, returned values, errors and timestamps, written as JSONL. This supports trajectory scoring, debugging and building training examples from successful runs.
Keep the answer key and grader out of the agent's reach. Agree the interface in the pilot so the same grader runs on every release, and version it alongside the schemas.
Deterministic reset and structured grader output make runs comparable across agent versions and let failures be traced to a specific step.
- Audience:
- Agent developers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How would a grader catch a duplicate record in a patient registration run?
- Which failure modes should an evaluation of set-up agents cover?
- How should we judge whether an agent set up an account correctly?
- How do we inspect generated set-up data for grounding and disclosure risk?
- What does state-based grading catch that reading the agent's reply would miss?
- How can I inspect the quality of a set-up environment before using it?