Skip to content
MirrorFirm

What should an evaluation team request inside a generated task pack?

A task pack specification should cover agent inputs, permitted actions and evaluator-only material, including cases where no record change is justified.

Specify these pack components together so each generated case has an executable task, a defined information boundary and enough evaluator material to support a defensible result.

- An agent-facing request that states the business concern without revealing the expected correction, preserving the need to investigate evidence and choose an action. - An initial record snapshot containing the relevant account status, outstanding requests and available supporting material at the moment the agent receives the task. - A role profile describing what the acting agent can inspect, propose and execute, with permissions expressed separately from the requester's preferred outcome. - A task-specific action catalogue explaining required arguments and observable results, including how rejected actions appear to the agent during the evaluation run. - Evaluator-only outcome constraints describing acceptable final states and forbidden changes, allowing several valid action sequences where the supplied policy permits them. - A rationale annotation connecting each expected decision to the evidence that supports it, enabling reviewers to challenge an answer key rather than trust it automatically. - Negative cases in which the suspected discrepancy is resolved without an adjustment, preventing the pack from teaching agents that every correction request requires a posting.

Define the contents needed to run and score a task before assessing the writing quality of individual records.

Audience:
Evaluation teams
Content type:
Checklist
Buyer question:
What can be generated

Content release: r-a220eb6e099b243b

More on Ledger adjustment