Skip to content
MirrorFirm

How can buyers check environment quality before scoring agents?

State invariants expose contradictions in generated complaint environments before those defects distort agent training, evaluation scores or interpretations of failure.

An environment can produce convincing records while allowing impossible business states. A case might reference an escalation that does not exist, record a response before receipt or show an investigation as complete without its required finding. Those defects can make agent behaviour appear wrong when the fixture itself is inconsistent.

Buyers can define invariants that must hold after every accepted transition. Every active obligation should belong to an existing case. Every completed investigation should satisfy its configured completion requirements. Every escalation outcome should reference the escalation it answers.

MirrorFirm-generated packages could be inspected against those rules before agent evaluation begins. Checks should distinguish deliberately inconsistent source evidence from accidental inconsistency in the environment's mechanics. Two witnesses disagreeing is a possible investigation challenge; a broken foreign key is a packaging defect.

Failure reports should identify the violated rule, affected records and transition that introduced the problem. Builders can then quarantine invalid episodes instead of silently counting them as agent failures. This creates a defensible boundary between the quality of the generated environment and the quality of the agent operating inside it.

Validate the environment's own logic before attributing unsuccessful episodes to an agent.

Audience:
Environment builders
Content type:
Explanation
Buyer question:
How to evaluate it

Content release: r-a220eb6e099b243b

More on Complaint escalation