How can evaluation hooks score several valid complaint-handling paths?
A state-based evaluator can accept different valid action sequences while detecting critical errors that a fluent final response or correct outcome might conceal.
Complaint handling may allow several valid sequences. A model might request a transaction check before drafting an acknowledgement, or prepare the acknowledgement first, provided the applicable obligations are met. Comparing the run with one exact action trace would penalise harmless variation.
An evaluator can instead use predicates over state and events. Checks might include `acknowledgement_recorded_before_due`, `required_findings_supported` and `handover_accepted_before_owner_release`. Each predicate should identify the evidence it consumes and the condition that satisfies it.
Critical errors need separate treatment. An unauthorised closure should remain visible even if a later action reopens the case and produces an acceptable final response. Event-level hooks can preserve these violations rather than scoring only the terminal snapshot.
A result record could contain `task_id`, `criterion_id`, `result`, `supporting_event_ids` and `review_reason`. Deterministic checks suit timestamps and permissions; evidence sufficiency may require a rubric and human adjudication. Buyers can ask for disagreement records when reviewers interpret a criterion differently. This produces scores that explain both valid variation and substantive failure without pretending every aspect of complaint quality is mechanically decidable.
Score obligations and prohibited transitions directly so legitimate process variation does not become a false failure.
- Audience:
- Model labs
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should developers run before training on generated complaint material?
- How can a task test escalation when an earlier case note is misleading?
- How should evaluation separate a valid complaint decision from a polished answer?
- Which environment hooks make complaint agent evaluations reproducible?
- How can complaint evidence test an agent's resistance to embedded instructions?
- How can evaluation hooks expose retry and concurrency failures?