Query-based segmentation models achieve strong performance and can run in real time on workstation GPUs. According to a new preprint on arXiv, the picture changes on edge GPUs: the transformer decoders in these models become deeply memory-bandwidth-bound. This bottleneck is invisible if one only tracks FLOPs, because the limiting factor is data movement, not arithmetic throughput.

The paper profiles query-based segmentation decoders on edge hardware and then applies acceleration techniques aimed at the memory bottleneck. The implication is that conventional compute-centric optimization may miss the real constraint on these devices. The abstract does not provide specific speedup numbers, but it points to memory bandwidth as the key factor for edge deployment.

For practitioners working on real-time segmentation in embedded settings, the takeaway is to measure memory traffic, not just FLOPs, when optimizing transformer decoders.