Which evaluation hooks show whether review follow-up was actually created?
Action replay and state assertions can distinguish a written promise of follow-up from a valid handover that exists in the environment.
A review agent may write that it has referred a finding without creating any work. Evaluation hooks can compare its final statement with persisted environment state. The relevant tool might be `create_follow_up`, returning a task identifier only after successful creation.
A generated task schema could include `task_id`, `source_review_id`, `finding_id`, `assigned_queue`, `required_action` and `status`. A valid handover requires the task to reference the correct finding and reach a queue permitted to handle it. Free-text mention of a team is insufficient.
An evaluator can replay the tool trace and assert that the task exists, that its references resolve and that the review's disposition matches the handover. It can also check for duplicate task creation after a tool timeout. An idempotency key can make retry behaviour explicit.
Reset should remove episode-created tasks and restore queue state before another run. Buyers can request a deliberately failed creation response to test whether the agent reports the failure accurately. This separates successful assessment from successful execution and exposes unsupported claims of completion.
Score operational follow-through by inspecting persisted tasks and routing, rather than relying on the agent's final message.
- Audience:
- Evaluation teams
- Content type:
- Integration
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What quality checks should precede training on generated review cases?
- How can a late event reveal whether an agent checks its assumptions before committing a decision?
- Which evaluation hooks make review runs reproducible and auditable?
- How should evaluation distinguish a sound review decision from a persuasive explanation?
- How can we tell whether an agent responds to meaningful evidence rather than superficial cues?
- How should buyers inspect realism and disclosure risk in generated review material?