How do I know whether my success criteria for booking tasks are strict enough?
A question-and-answer on testing grader strictness with deliberately wrong runs, so criteria reject plausible but incorrect booking outcomes.
**What is the quickest test of a grader?**
Write a few deliberately wrong runs and score them. For booking, useful ones are: activating before security is confirmed, booking the old margin after an amendment, choosing the wrong repayment frequency, and silently skipping a failed condition. A strict grader fails all of them with distinct causes.
**Should I grade the path or the end state?**
Grade the end state first, because it is what servicing inherits. Add path checks only for rules that cannot be seen in the final records, such as whether the agent consulted the policy or the amended approval.
**How should exceptions be scored?**
When the inputs do not support booking, a returned file with the right reason code and owner role is a pass. Booking anyway is a fail, even if the facility looks tidy.
**How do I avoid a grader that is too brittle?**
Accept harmless variation, such as note wording or the order of independent steps, and be exact on amounts, dates, rates, statuses and links.
**Can a task pack include the bad runs?**
A pack could include reference passing and failing trajectories, and you can ask for them in a pilot.
Run known-bad trajectories through your grader first. If it passes them, tighten the criteria before judging any agent.
- Audience:
- Environment builders
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- Which failure modes should I probe when evaluating agents on facility booking?