Skip to content
MirrorFirm

What scoring hooks can evaluate requests and final case state?

An evaluation harness can score gap detection by set comparison, request quality by item coverage, and agent runs by comparing final case state with a reference state.

Three scoring layers cover most uses, and each works from fields already in the records.

**1. Gap detection.** Compare predicted gaps with reference gaps by `(code, rule_id, doc_id)`. Report precision and recall separately, because over-asking and under-asking are different failures.

**2. Request quality.**

- coverage: every reference gap is named in the message - no extras: nothing is requested that the policy does not require - form: the acceptable document form is stated - consequence: the lapse outcome is stated

The first two can be checked by structure if the model returns item lists with its message. The last two usually need a rubric or a judge model, which should be validated against human reading on a small sample.

**3. Final state in an environment.** An agent works through tool actions such as `get_case`, `get_policy`, `read_document`, `set_status` and `send_request`. At the end, compare the case status, the request record and the audit log against a reference. Penalise status changes that were not justified by a gap.

**Reset.** Each episode should start from a seeded snapshot, and a reset call should restore it. Log the seed and release version with every result.

Share the scoring code and the schema version with customers if you license the pack onward, so they report comparable results.

All content is generated. Confirm which hooks a given pack supports in a sample.

Scoring on structure first, with text judgement second, gives your customers repeatable results and cheaper evaluation runs.

Audience:
Licensed resellers
Content type:
Integration
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Document sufficiency