What are the success criteria and common failure modes for facility recording agents?

**What counts as success?** The final state matches the expected state on every graded field, no forbidden action was taken, and any required hold or query was raised to the correct role. Partial credit can be defined per field.

**What are the common failure modes?** - Booking from the approval when the offer differs - Releasing a drawdown with an open condition - Linking a party with a similar name instead of the right entity - Setting derived fields, such as schedule or fee basis, from defaults - Stopping to ask when the policy lets the agent proceed - Reporting completion when the state is invalid

**Is over-escalation a failure?** Yes. A case set should include clean cases where the right answer is to book, so a model that holds everything does not score well.

**How do I tell if differences between models are meaningful?** Run each case with several seeds and repeat attempts, and inspect variation per case instead of relying on one aggregate score. Ask for case-level labels so you can slice by exception type.

**Can I use the grader for training?** A grader can produce outcome rewards, but check whether it can be gamed, for example by writing the expected values directly. Hold out a set of cases never used for training.

**What if my grader disagrees with the supplied one?** Treat the disagreement as a finding. Ask for the policy basis of the expected outcome.