What does a quality defect in a sample look like, and what do I do?
A marketplace data engineer finds two current repayment schedules on one facility after an amended approval, reports the defect, and adds the rule to their own intake checks.
A data engineer at a marketplace receives a sample of facility records and runs a reconciliation script before listing. For FAC-7712, a trade finance facility with a limit of $900,000, the repayment schedule rows sum to a principal larger than the limit.
The engineer traces the rows to schedule SCH-7712-A and finds a second schedule, SCH-7712-B, generated after an amended approval reduced the limit. Both schedules are marked current. A servicing agent reading this would collect against the older, larger schedule.
The engineer records the finding with the facility ID, the two schedule IDs and the rule broken: only one current schedule per facility. The note goes back to MirrorFirm as a defect report. In the revised sample the older schedule carries `superseded` status and the amended approval is linked by `supersedes_approval_id`.
The engineer then adds the one-current-schedule rule to the marketplace's own intake checks so that every later release is tested the same way.
If the defect had gone unnoticed, a customer's evaluation would have scored an agent as correct for following a schedule that should never have been live, and the marketplace would have carried the support burden.
**What it teaches:** amendments and supersession are where generated data most often drifts. Test them directly, and keep your own checks.
Amendments are where generated data drifts most, so test supersession directly and keep your own intake checks rather than relying only on the supplier's.
- Audience:
- Licensed resellers
- Content type:
- Scenario
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- How do I know whether my success criteria for booking tasks are strict enough?