Skip to content
MirrorFirm

What success criteria and review questions should I use on a set-up task pack?

Answers questions on defining success, partial credit, detecting tasks that are too easy, handling valid alternative paths and reviewing grounding and disclosure risk in generated samples.

**What counts as success on a task?** The episode ends in the expected state, or in one of the listed acceptable variants, with no forbidden writes. A text reply claiming success does not count by itself.

**Can I give partial credit?** Yes. Grade required records, correct links, rule compliance and unintended writes as separate components, then combine them as suits your training or reporting needs.

**How do I tell whether tasks are too easy?** Run a naive baseline that creates records straight from the approval without checking anything. If it passes most tasks, the exceptions are not biting. Tasks where the naive path leads to a downstream consequence, such as a wrong invoice or a contested claim, are harder to pass by accident.

**What if my agent finds a valid path the grader rejects?** Treat it as a grader defect, not an agent failure. Ask how acceptable variants are expressed and whether you can add them. Keep a record of such cases for the next release.

**How do I check grounding and disclosure risk?** Ask how the structure and exceptions were informed, then review samples for names, identifiers or combinations that look like they might belong to a real party. Check that rare attributes are not unusually specific. This is a review you carry out; generation does not replace it.

**Which failure modes should I report separately?** Missing actions, wrong values, duplicate creation, unintended writes and ignored conditions.

Define success as an end state with no forbidden writes, and treat grader rejections of valid paths as defects to report back.

Audience:
Environment builders
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Account setup