Ambivalence and hesitancy are subtle states that show up in both what people say and how they say it. A new paper from researchers TingTing Gu, JinCheng Luo, and WenQiu Guo proposes a framework that combines full-video transcripts with speech waveforms to recognise these states at the video level. The system encodes text with pre-trained multilingual BERT and audio with wav2vec 2.0, then projects both into a shared representation space.
The core of the approach is a pair of stacked bidirectional cross-attention blocks. These allow linguistic tokens to attend to acoustic frames and acoustic frames to attend to linguistic context, without needing explicit alignment between the two modalities. After aggregation via masked mean pooling, the model combines the resulting sequences using concatenation, element-wise products, and absolute differences for binary classification.
Evaluated on the subject-independent split of the Behavioural Ambivalence/Hesitancy (BAH) dataset, the system reports 72.19% accuracy, 75.00% precision, 81.13% recall, 77.95% A/H F1-score, and 70.16% Macro-F1. The results suggest that jointly modelling linguistic and acoustic information can help automatically identify these subtle behavioural cues, though the paper does not compare against unimodal baselines or other fusion strategies in the provided abstract.