Feature Decoupling with Modality Modulation for Multimodal Sentiment Analysis

doi:10.1007/978-981-95-3393-0_8

Feature Decoupling with Modality Modulation for Multimodal Sentiment Analysis

2026-01-01

0

PRE

AI

DOI：10.1007/978-981-95-3393-0_8

Original

Original request for help

Save

Abstract

En 中文

Multimodal sentiment analysis aims to extract and integrate meaningful information from diverse modalities to infer a speaker's emotional state. Due to the inherent heterogeneity among modalities, most existing approaches decouple modalities into specific and invariant features, which partially capture cross-modal representations. However, in multimodal tasks, certain modalities often dominate the optimization process, leading to the under-optimization of weaker modalities. To address this imbalance, we propose the Modal Modulation Adaptive Fusion Network (MMAFNet), which enhances the learning of valuable information from each modality. Specifically, for modality-specific features, we introduce a gradient modulation strategy that dynamically adjusts learning rates to prioritize weaker modalities. For modality-invariant features, we employ a parameter reset strategy based on intermodal distances to mitigate overfitting and strengthen feature extraction in underperforming modalities. Additionally, an adaptive fusion module combines modality-specific and invariant features according to their learned weights. Our comprehensive analysis of feature characteristics and tailored modulation strategies effectively alleviates modality imbalance. Extensive experiments on two benchmark datasets demonstrate the superiority of our approach.

Keywords:

Multimodal sentiment analysis

modality modulation

modality imbalance

Feature Decoupling with Modality Modulation for Multimodal Sentiment Analysis

Abstract

Journal

Organization