What scoring hooks can evaluate requests and final case state?
An evaluation harness can score gap detection by set comparison, request quality by item coverage, and agent runs by comparing final case state with a reference state.
Three scoring layers cover most uses, and each works from fields already in the records.
**1. Gap detection.** Compare predicted gaps with reference gaps by `(code, rule_id, doc_id)`. Report precision and recall separately, because over-asking and under-asking are different failures.
**2. Request quality.**
- coverage: every reference gap is named in the message - no extras: nothing is requested that the policy does not require - form: the acceptable document form is stated - consequence: the lapse outcome is stated
The first two can be checked by structure if the model returns item lists with its message. The last two usually need a rubric or a judge model, which should be validated against human reading on a small sample.
**3. Final state in an environment.** An agent works through tool actions such as `get_case`, `get_policy`, `read_document`, `set_status` and `send_request`. At the end, compare the case status, the request record and the audit log against a reference. Penalise status changes that were not justified by a gap.
**Reset.** Each episode should start from a seeded snapshot, and a reset call should restore it. Log the seed and release version with every result.
Share the scoring code and the schema version with customers if you license the pack onward, so they report comparable results.
All content is generated. Confirm which hooks a given pack supports in a sample.
Scoring on structure first, with text judgement second, gives your customers repeatable results and cheaper evaluation runs.
- Audience:
- Licensed resellers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What evaluation hooks should a sufficiency task pack expose to my test harness?
- How does a scored evaluation run expose a missed sufficiency check?
- Which sufficiency failure modes should I test before deploying an agent?
- How can I inspect the quality of generated sufficiency cases before using them?
- What success criteria should I use for an agent that judges document sufficiency?
- Which failure modes should we score separately on sufficiency tasks?