.png){kind=icon}
.png){kind=icon}
Return
Affection-Guided Bottleneck Diffusion for Missing Modality Issue in Multimodal Affective Computing
.png){kind=icon}
PRE
AI
DOI：10.1109/TAFFC.2025.3632063.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Missing modality issue in multimodal affective computing severely hinders the robustness and performance of multimodal learning, particularly in real-world scenarios. Existing methods often fail in fully exploiting the remaining modalities, leading to noisy reconstruction process for the missing modalities and yielding suboptimal results. Besides, most of these methods rely on designing sophisticated networks to handle various missing scenarios, which prevents them from taking advantage of the original pre-trained multimodal networks trained for complete multimodal inputs. To address these challenges, we propose Affection-guided Bottleneck Diffusion (ABDiff), a novel approach leveraging score-based diffusion generative encoders to reconstruct missing modalities in the latent space without modification to the pre-trained fusion models. By incorporating self- and cross-attention mechanisms inside and among the missing and remaining modalities, ABDiff captures both modality-specific dynamics and cross-modal interactions during generation. Furthermore, an affection-guided information bottleneck is introduced to filter task-unrelated noise and modality-specific redundancy, stabilizing the generation process of missing modalities. The generated representations are seamlessly integrated with the remaining modalities into the pre-trained fusion networks. Extensive experiments on four public multimodal affective computing datasets demonstrate that ABDiff surpasses previous methods under both complete and incomplete modality scenarios.
Keywords:
Multimodal affective computing
missing modality issue
diffusion model
conditional generation
information bottleneck
Journal
IF:
9.8
Papers:
1.3K
Citations:
9.1K
Organization
View more organizations