How should we assess grounding and disclosure risk in a generated booking sample?
Ask for documentation on how content was generated and grounded, run overlap and identifier checks yourself, and treat the results as evidence to weigh, not as a guarantee.
**Is the data real?** No. MirrorFirm output is generated and does not contain real business records. It is informed by how businesses operate, which is a different thing from copying their data.
**Does generated mean risk-free?** No. Generation does not guarantee anonymity, privacy compliance or regulatory approval. Treat disclosure as a risk to inspect, not a property you can assume.
**What can we check ourselves?** - Search for identifiers that look like real ABNs, account numbers or addresses and test whether they resolve to real entities. - Compare free-text fields against public sources and your own internal documents for near-copies. - Look for rare combinations, such as an unusual product with a distinctive amount and date, that could point to a single real case. - Check that names of organisations in the data are generated or are Aster Financial.
**What should we ask the provider?** How content was produced, what grounding material informed it, what overlap checks were run, and what the process is if a reviewer finds a concern.
**What about realism?** Have an operations reviewer assess whether conditions, handover notes and exception reasons read as a practitioner would write them. Record disagreements and feed them back during the pilot.
Independent checks on a sample give you your own evidence on realism and disclosure risk instead of relying on assurances.
- Audience:
- Evaluation teams
- Content type:
- FAQ
- Buyer question:
- How to evaluate it
Content release: r-a220eb6e099b243b
Related questions
- What checks should run on every booking task in an evaluation?
- Which failure modes should we expect to see and how should we score them?
- How can a buyer inspect the quality of generated booking material?
- What reset, snapshot and grader hooks would an evaluation harness need?
- How would an evaluation expose an agent that creates duplicate accounts on retry?
- How do I know whether my success criteria for booking tasks are strict enough?