Multimodal emotion recognition systems typically fuse facial expressions, vocal prosody, and other signals with softmax classifiers. These models tend to output high-confidence predictions even when the input is corrupted or the modalities contradict each other, such as a smiling face paired with an angry tone of voice.
The paper, an arXiv preprint, proposes an evidential multimodal transformer that instead treats each modality as a source of evidence. It applies Dempster-Shafer fusion to combine these sources, allowing the model to represent uncertainty explicitly rather than forcing a single overconfident label.
This design is particularly relevant for real-world applications where sensors are noisy or ambiguous. By acknowledging disagreement between modalities, the method aims to make emotion recognition more robust than standard softmax-based approaches.