A new arXiv paper proposes a six-stage framework for auditing how reproducible scientific claims are across a body of computer science research, and applies it for the first time to the neuro-symbolic AI (NSAI) literature. The framework is designed as a general tool, with this audit serving as its first demonstration.
The audit finds that only 6.5% of the neuro-symbolic literature can be reproduced from its published artifacts. In other words, the vast majority of claims in this area do not hold up when researchers rely solely on what authors have released.
The result quantifies a known but rarely measured problem: published papers in AI often lack the code, data, and detailed procedures needed to verify their results. By making the audit process explicit and staged, the authors offer a way to measure reproducibility gaps systematically in other subfields as well.