.png){kind=icon}
.png){kind=icon}
Return
Prototype-Based Modality-Specific Feature Distillation for Incomplete Multimodal Sentiment Analysis
.png){kind=icon}
PRE
AI
DOI：10.1109/TETCI.2025.3641665.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Incomplete multimodal sentiment analysis is to recognize human emotional states by leveraging the incomplete multimodal information including text, audio, and visual data. Existing methods mainly focus on reconstructing from available modalities, while ignoring the specific information of missing modalities and the inconsistent distribution between original and recovered data. In this work, we propose a novel Prototype-based modality-specific feature Distillation Network called ProDNet, which can recover the specific information of missing modalities with more consistent distributions between original and recovered data. In particular, we design a category-specific prototype for each modality to store the modality-specific information, and reconstruct the missing modality by combining the invariant features from existing modalities with the prototype of the missing modality in a feature self-distillation manner. To obtain favorable invariant features, we design an invariant feature distillation strategy, in which the weight-gating mechanism is introduced to enhance invariant feature representation. Finally, to reduce distributional inconsistency, we design a distribution consistency distillation loss to enforce the consistent distribution of original and recovered data by considering the sample correlation between teacher and student networks. Extensive experiments on two benchmark datasets, CMU-MOSI and CMU-MOSEI, demonstrate the superior performance of the proposed method, particularly in scenarios with higher missing rates.
Keywords:
Multimodal sentiment analysis
missing modalities
knowledge distillation
Journal
I
IF:
6.5
Papers:
1.4K
Citations:
4.5K
Organization
View more organizations