Token merging has become a popular way to reduce the computational cost of Vision Transformers, by grouping redundant tokens before they pass through the network. According to the paper, most existing methods decide which tokens to merge based on feature similarity alone. That works well for semantic tasks, where similar features often correspond to the same concept.

The authors introduce MaskMerge, a structure-aware approach that incorporates structural information when selecting tokens to merge. The motivation is that instance segmentation requires distinguishing individual objects, not just semantic categories, so merging purely by feature similarity may blur object boundaries or merge distinct instances.

The abstract indicates that this structural criterion is the key departure from prior work. While the full details are not provided in the abstract, the paper positions MaskMerge as a way to make token merging more effective for instance segmentation, potentially offering efficiency gains without sacrificing the precision needed for separating overlapping objects.