Referring expression segmentation (RES) typically relies on typed text to tell a model which object to pick out in an image. That assumption becomes awkward for unmanned aerial vehicles (UAVs), where operators often need hands-free, rapid communication. The authors of this paper argue that raw audio is a more natural and information-rich instruction channel than transcribed text, and they build a system that works directly from sound.
To support this shift, they constructed Aerial-AudioRES by augmenting the VisDrone dataset with synthesized speech, forming audio-text-image-mask quadruples. They added an ambiguity filtering mechanism to ensure each spoken instruction uniquely identifies a target, and included eight timbres plus four realistic noise types to mimic real operating conditions. Their proposed network, ARES-Net, learns a soft alignment between continuous acoustic representations and multi-scale visual features, bypassing the discrete text-matching step used by prior models.
In experiments, ARES-Net reached a peak accuracy of 64.23% in overall intersection-over-union (oIoU) and generalized well on a collected real-human test set. More striking, its total inference latency was one-third that of a comparable text-based method, suggesting audio commands can enable faster drone response. The code is publicly available, allowing others to build on the audio-centric paradigm for aerial scene understanding.