How should I inspect the quality of generated schedule state?
Quality can be inspected through arithmetic invariants, cross-record consistency, policy-to-outcome agreement, variety across cases and a separate review of grounding and disclosure risk.
Start with checks that need no model at all. Schedule data is unusually easy to test because it must obey arithmetic and ordering rules.
**Arithmetic invariants.** Instalments should sum to the plan principal plus fees. Allocated payments should not exceed amounts received. Balances should equal the sum of what remains.
**Ordering and dates.** Sequence numbers should match due-date order. No instalment should fall after the plan end date, and no due date should fall before the plan start.
**Link integrity.** Every foreign key should resolve. Every event should reference records that exist in the before and after states.
**Policy agreement.** Take recorded actions and test whether they were permitted under the policy version in force. Approvals should appear where thresholds require them.
**Variety.** Look at the spread of plan lengths, amounts, statuses and exception types. A set where every plan is regular will not test anything.
**Grounding and disclosure.** Because the output is generated, ask how its structure was informed and review whether any record looks like a copy of a real source. Do not assume generation removes that risk.
Run the first four as scripts on the sample, then run a baseline agent to see whether tasks are solvable and whether the grader can tell good from poor behaviour.
Most quality problems can be found by automated checks on the records themselves, before any agent is run against them.
- Audience:
- Environment builders
- Content type:
- Explanation
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- How can a grader catch a quiet error in a premium instalment change?
- How do I check grounding and disclosure risk in generated data?
- How do I judge the quality of generated schedule data?
- How do state diffs, graders and hooks fit into a schedule evaluation?
- What success criteria should I set for a schedule agent evaluation?
- How does an evaluator catch rounding drift when an agent splits an instalment plan?