The DroneCATS benchmark, introduced in a new preprint, tests whether multimodal large language models (MLLMs) can act as generalist vision-language-action agents for drone control. Unlike prior evaluations, which mostly used simulation, a single drone, and static targets, DroneCATS varies target visibility, motion, scene, and simulator, and extends to real drones. It also includes multi-drone suites: one model driving several drones, and a commander model directing one sub-agent per drone.

Across ten MLLMs, from frontier models to small open models, the authors found that frontier models achieved the highest success rates, but their latency caused them to lose moving targets. Small open models often failed to declare arrival even when they reached the target. Guided by this failure analysis, the team fine-tuned a 2B model on drone action data synthesized from public RGB-D images; its success rate rose from 0% to 31% in simulation and outperformed the frontier models in real flight.

In the multi-drone settings, every zero-shot model succeeded more often as the commander of one sub-agent per drone than when it drove all drones from one context, and commanding success followed general vision-language model ability. On real drones, results broadly followed those in simulation, though latency and hardware limits mattered more. The benchmark is set to be fully open-sourced.