Enterprises are increasingly deploying AI agents for IT operations, but these agents often fail in unpredictable ways. A new collaboration between IBM and UC Berkeley addresses this by introducing two tools: IT-Bench, a benchmark for enterprise IT scenarios, and MAST, a framework for diagnosing multi-agent system behavior. Together, they offer a structured way to evaluate where and why agents break down.
The work is significant because it moves beyond anecdotal reports of agent failures toward reproducible, measurable diagnostics. By using IT-Bench to stress-test agents on realistic tasks and MAST to trace errors back to their root causes, developers can identify patterns such as miscommunication between agents or incorrect tool usage.
While the source does not provide specific failure statistics, it highlights the need for systematic evaluation in enterprise settings. The authors argue that without such benchmarks and diagnostic frameworks, improving agent reliability remains guesswork. This research provides a foundation for building more robust AI systems in complex operational environments.