.png){kind=icon}
Return
SHF-MSA: A Hypergraph-Based Multimodal Sentiment Analysis Model
.png){kind=icon}
PRE
AI
DOI：10.1109/taffc.2026.3722051.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Reliable recognition of human emotions from multimodal signals is important for human-computer interaction and affective intelligence systems. However, multimodal sentiment analysis remains challenged by heterogeneous modality representations, insufficient modeling of high-order interactions, and limited generalization under distribution shifts. To address these issues, this paper proposes SHF-MSA, a hypergraph-based multimodal sentiment analysis model that integrates representation coordination, structured high-order fusion, and multi-task learning. First, a Unified Pre-encoding Layer (UPL) employs modality-specific encoding, shared-space projection, and statistical normalization to reduce representation discrepancies among textual, acoustic, and visual modalities while preserving modality-specific and temporal information. Next, a Temporal Synergistic Hypergraph Encoding (TSHE) module constructs cross-modal hyperedges over the same word-level positions and intra-modal temporal-window hyperedges to jointly capture cross-modal co-occurrence and local temporal dependencies. Node-hyperedge-node propagation is further combined with instance dependent hyperedge routing to adaptively reweight hyperedge propagation according to sample-specific evidence. Finally, a Multi-task Fusion Output Layer (MFOL) aggregates structured representations across temporal and modality dimensions and jointly performs sentiment regression, polarity classification, and seven-class sentiment-level prediction. Experiments on CMU-MOSI, CMU-MOSEI, and their out-of-distribution splits demonstrate that SHF-MSA achieves competitive performance across both regression and classification tasks, with particularly strong results in strict polarity discrimination and distribution-shift evaluation.
Keywords:
Multimodal sentiment analysis
multimodal fusion
hypergraph learning
multi-task learning
Journal
IF:
9.8
Papers:
1.4K
Citations:
9.1K
Organization
Cited Papers
No cited papers available