Skip to content
MirrorFirm

How do I know whether my success criteria for booking tasks are strict enough?

A question-and-answer on testing grader strictness with deliberately wrong runs, so criteria reject plausible but incorrect booking outcomes.

**What is the quickest test of a grader?**

Write a few deliberately wrong runs and score them. For booking, useful ones are: activating before security is confirmed, booking the old margin after an amendment, choosing the wrong repayment frequency, and silently skipping a failed condition. A strict grader fails all of them with distinct causes.

**Should I grade the path or the end state?**

Grade the end state first, because it is what servicing inherits. Add path checks only for rules that cannot be seen in the final records, such as whether the agent consulted the policy or the amended approval.

**How should exceptions be scored?**

When the inputs do not support booking, a returned file with the right reason code and owner role is a pass. Booking anyway is a fail, even if the facility looks tidy.

**How do I avoid a grader that is too brittle?**

Accept harmless variation, such as note wording or the order of independent steps, and be exact on amounts, dates, rates, statuses and links.

**Can a task pack include the bad runs?**

A pack could include reference passing and failing trajectories, and you can ask for them in a pilot.

Run known-bad trajectories through your grader first. If it passes them, tighten the criteria before judging any agent.

Audience:
Environment builders
Content type:
FAQ
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Loan booking