Affection-Guided Bottleneck Diffusion for Missing Modality Issue in Multimodal Affective Computing

doi:10.1109/TAFFC.2025.3632063

Return

Affection-Guided Bottleneck Diffusion for Missing Modality Issue in Multimodal Affective Computing

2025-11-12

PRE

DOI：10.1109/TAFFC.2025.3632063

Original

Original request for help

Save

Abstract

En 中文

Missing modality issue in multimodal affective computing severely hinders the robustness and performance of multimodal learning, particularly in real-world scenarios. Existing methods often fail in fully exploiting the remaining modalities, leading to noisy reconstruction process for the missing modalities and yielding suboptimal results. Besides, most of these methods rely on designing sophisticated networks to handle various missing scenarios, which prevents them from taking advantage of the original pre-trained multimodal networks trained for complete multimodal inputs. To address these challenges, we propose Affection-guided Bottleneck Diffusion (ABDiff), a novel approach leveraging score-based diffusion generative encoders to reconstruct missing modalities in the latent space without modification to the pre-trained fusion models. By incorporating self- and cross-attention mechanisms inside and among the missing and remaining modalities, ABDiff captures both modality-specific dynamics and cross-modal interactions during generation. Furthermore, an affection-guided information bottleneck is introduced to filter task-unrelated noise and modality-specific redundancy, stabilizing the generation process of missing modalities. The generated representations are seamlessly integrated with the remaining modalities into the pre-trained fusion networks. Extensive experiments on four public multimodal affective computing datasets demonstrate that ABDiff surpasses previous methods under both complete and incomplete modality scenarios.

Keywords:

Multimodal affective computing

missing modality issue

diffusion model

conditional generation

information bottleneck