Must an agent follow one reference sequence to pass?
Complaint evaluations can accept different investigation paths when they satisfy the same obligations, respect permissions and support the eventual decision with available evidence.
**Should every episode have one correct action sequence?**
Usually the stronger design specifies required dependencies and acceptable outcomes. An agent may review the transaction history before reading correspondence, or reverse that order, without changing the quality of its investigation.
**What should remain mandatory?**
Some ordering constraints are substantive. Required evidence must be obtained before a finding relies on it. A remedy requiring approval must not be executed before approval exists. These dependencies belong in the scoring rules even when other steps can vary.
**How can a buyer inspect the scoring approach?**
Ask for accepted alternative traces and rejected traces that explain the boundary. A task pack could include a direct resolution path, a justified specialist escalation and a path that remains unresolved because required information cannot be obtained.
MirrorFirm could generate such alternatives with their supporting records. Builders should check that the evaluator does not reward unnecessary actions simply because they resemble the reference trace. It should also avoid penalising a justified unresolved outcome when the environment provides no valid route to certainty. The target is defensible process execution, not imitation of a preferred script.
Score justified outcomes and required dependencies without overfitting evaluation to one scripted sequence.
- Audience:
- Environment builders
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should developers run before training on generated complaint material?
- How can a task test escalation when an earlier case note is misleading?
- How should evaluation separate a valid complaint decision from a polished answer?
- Which environment hooks make complaint agent evaluations reproducible?
- How can complaint evidence test an agent's resistance to embedded instructions?
- How can evaluation hooks expose retry and concurrency failures?