How should follow-up quality be scored when several responses are acceptable?
A useful rubric separates mandatory business constraints from flexible wording, allowing different valid responses without accepting incorrect commitments or missing information.
**Does a correct response need to match a reference message?**
No. Several messages can satisfy the same business requirement. Exact wording comparisons can penalise valid responses while overlooking a polished message that contains an unsupported claim.
**What should the rubric check?**
Define required meaning: the outstanding item, the requested next step and the correct route for responding. Define forbidden meaning separately, such as asserting that a penalty has been applied when the record contains no such event. Where the policy requires a particular notice, identify that requirement explicitly.
**How should partial success be handled?**
Separate action eligibility, recipient selection and message content. A well-written message to an unauthorised recipient should not pass because its tone is appropriate. Likewise, a correctly routed message can still omit the information needed to resolve the delay.
**What could MirrorFirm generate?**
A task pack could include acceptable semantic variants, disallowed assertions and evidence references for each scoring condition. Developers can use these to inspect automated grading disagreements and distinguish genuine business errors from harmless differences in phrasing.
Score semantic obligations and prohibited claims so evaluation remains useful across varied agent writing styles.
- Audience:
- Agent developers
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should buyers run before using a generated task pack?
- Which evaluation hooks expose duplicate actions and unreliable recovery?
- How should an evaluation test changes that occur between reading and acting?
- How can buyers tell whether an agent follows policy instead of superficial cues?
- What should we inspect before trusting an environment's evaluation results?
- How can we tell whether an agent learned the policy or a dataset shortcut?