Skip to content
MirrorFirm

Can a release include mismatch cases that have no single correct cause?

A task pack can mix determinable, ambiguous and undeterminable cases, with acceptable end states recorded as sets and rules rather than a single answer key.

**Q: Can a release include cases where the cause cannot be fully determined?**

Yes, and it should. A task pack could mix three kinds of case so the agent is tested on judgement, not only on diagnosis.

- **Determinable.** The evidence in the environment points to one cause and one correct fix. - **Ambiguous.** Two causes fit the evidence. The correct behaviour is to take the reversible action or ask the counterparty, and to say what is still unknown. - **Undeterminable from available data.** The right end state is a documented escalation with the evidence gathered so far.

**Q: How is the expected answer recorded for an ambiguous case?**

As a set of acceptable end states plus a list of unacceptable ones, not a single answer key. A buyer can ask for each case to carry a rationale explaining why each state sits where it does.

**Q: Why include these at all?**

An agent trained or tested only on solvable cases is never checked for inventing a cause. Cases with no clean answer give you a way to measure whether an agent makes one up, and whether it escalates when it should.

**Q: Does that make grading subjective?**

It makes it rule-based rather than label-based. The grader checks properties of the end state and the trace, such as 'no irreversible write before evidence was gathered', which can be run automatically. Scope the rule set in a pilot and review it with your own domain staff.

Cases without a clean answer let you measure whether an agent invents a cause or escalates when the evidence runs out.

Audience:
Environment builders
Content type:
FAQ
Buyer question:
What can be generated

Content release: r-a220eb6e099b243b

More on Reconciliation mismatch