Multimodal learning often faces incomplete data, where some modalities are missing. Recent Mixture-of-Experts (MoE) approaches have tried to address this in two ways: routing each missingness pattern to a dedicated expert, or imputing the missing modality from the available ones. The first approach scales poorly as the number of modalities grows, while the second is only mentioned in the abstract without further detail.
The new paper introduces TWIN-MoE, a two-stage intra-expert imputation network. Rather than assigning a separate expert to every possible pattern of missing data, the method performs imputation inside each expert in two stages. This is intended to reduce the scaling burden while still handling incomplete inputs.
The abstract does not specify the exact mechanics of the two stages or how the imputation is trained. It only states the problem with existing methods and the name of the proposed solution. Further details would require reading the full paper.