As LLM-based agents are deployed in unsupervised settings, the need for rigorous testing grows. The authors of a new arXiv paper argue that existing approaches rely on acceptance-level evaluation from the user's perspective, which requires manual review, resists automation, and makes it difficult to trace failures back to specific components. They propose a structural testing framework that operates at a deeper technical level.
The framework captures agent trajectories using OpenTelemetry traces, enforces reproducible LLM behavior through mocking, and automates verification with assertions. This allows agents to be tested component-by-component within automated workflows. The authors demonstrate how this enables established software engineering practices—such as the test automation pyramid, regression testing, test-driven development, and multi-language testing—to be adapted for agent systems.
In representative case studies, the structural approach delivered automated execution and faster root-cause analysis compared with acceptance testing. The authors report reduced testing costs and improved agent quality through higher coverage, reusability, and earlier defect detection. An open source reference implementation is available on GitHub, and the paper has been accepted for publication at the 2025 IEEE International Conference on Big Data.