Skip to content
MirrorFirm

How can evaluation hooks score several valid complaint-handling paths?

A state-based evaluator can accept different valid action sequences while detecting critical errors that a fluent final response or correct outcome might conceal.

Complaint handling may allow several valid sequences. A model might request a transaction check before drafting an acknowledgement, or prepare the acknowledgement first, provided the applicable obligations are met. Comparing the run with one exact action trace would penalise harmless variation.

An evaluator can instead use predicates over state and events. Checks might include `acknowledgement_recorded_before_due`, `required_findings_supported` and `handover_accepted_before_owner_release`. Each predicate should identify the evidence it consumes and the condition that satisfies it.

Critical errors need separate treatment. An unauthorised closure should remain visible even if a later action reopens the case and produces an acceptable final response. Event-level hooks can preserve these violations rather than scoring only the terminal snapshot.

A result record could contain `task_id`, `criterion_id`, `result`, `supporting_event_ids` and `review_reason`. Deterministic checks suit timestamps and permissions; evidence sufficiency may require a rubric and human adjudication. Buyers can ask for disagreement records when reviewers interpret a criterion differently. This produces scores that explain both valid variation and substantive failure without pretending every aspect of complaint quality is mechanically decidable.

Score obligations and prohibited transitions directly so legitimate process variation does not become a false failure.

Audience:
Model labs
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Complaint escalation