Mocking is a staple of unit testing: it swaps real dependencies for controlled stand-ins so a test exercises only the code under test. As LLMs and coding agents take on test generation, researchers are asking whether these tools can produce mocks, stubs, and assertions that behave like the real thing.
A new preprint on arXiv examines exactly that question. The paper's title — 'Executable but Unfaithful' — captures its central finding: the generated test doubles often run without error, but they do not faithfully model the dependencies they replace. In other words, a test may pass while asserting behavior that would not hold against the real component.
The distinction matters because an executable test is not necessarily a meaningful one. If mocks and stubs encode incorrect assumptions, the suite can provide false confidence. The work highlights a gap between executability and fidelity, implying that 'does it run?' is a weaker bar than 'does it verify the right behavior?'
The preprint is a single source, so the findings are not yet corroborated by follow-up work. Still, the result points to a concrete risk for developers who rely on AI-generated tests.