How can evaluation hooks check whether an agent respects evidence access limits?
Access-aware evaluation can inspect tool calls and evidence citations, testing whether an agent builds its pack using only records available to its role.
An environment could expose `search_records`, `read_record`, `request_access` and `submit_pack` actions. Each call should execute under an explicit role and return an access outcome. A search result may reveal that a restricted record exists without exposing its contents.
The action log could record `run_id`, `actor_role`, `tool_name`, `record_id`, `access_decision` and `state_revision`. Submitted citations should reference record identifiers so an evaluator can compare the final pack with records successfully read during the run.
Evaluation hooks could flag claims based on inaccessible content, attempts to change roles without authorisation and unsupported statements that access was granted. They could also credit an appropriate access request followed by a qualified finding when the request remains unresolved.
Reset should restore permissions, pending requests and case state. A reproducible seed alone is insufficient if an earlier run's access grant persists into the next run.
Licensed resellers could package the permission model, action contract and trace schema with the environment. Customers could then inspect both the final answer and the route used to obtain its evidence, rather than treating a correct-looking pack as sufficient.
Measure access behaviour alongside answer quality so customers can identify packs assembled through impermissible routes.
- Audience:
- Licensed resellers
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- Which evaluation hooks reveal how an agent assembled its evidence?
- How can an evidence task test resistance to instructions inside retrieved records?
- Which quality checks reveal shortcuts that could undermine an evidence dataset?
- Must an agent's assessment match one reference answer exactly?
- How do you evaluate whether an agent selected the right cases for review?
- What quality checks should precede an agent evaluation?