A new arXiv preprint introduces SANZARU, a benchmark designed to probe a blind spot in language-model evaluation: how well models cooperate when they do not all have the same information or communication abilities. The authors argue that standard benchmarks measure individual competence, but real-world deployment often requires teams to coordinate under asymmetric conditions.
SANZARU adapts cooperative bomb-defusal puzzles to create controlled scenarios where team members hold different knowledge and have limited or uneven channels for communication. The benchmark is intended to expose whether models can form shared protocols on the fly, rather than relying on pre-existing coordination.
The paper's central finding, as stated in the abstract, is that a model's ability to solve a task alone does not ensure that a team of models can solve it together. This is a single preprint, so the results have not yet been peer-reviewed, but the benchmark points to an important distinction between individual capability and collective performance in multi-agent AI systems.