.png){kind=icon}
.png){kind=icon}
Return
Efficient Long-distance Latent Relation-aware Graph Neural Network for Multi-modal Federated Emotion Recognition
.png){kind=icon}
PRE
AI
Y
W
J
T
H
DOI：10.1016/j.jpdc.2026.105269.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The task of multimodal emotion recognition (MER) is to analyze input data (e.g., text, video, and audio) and identify and predict the emotional state expressed by the multimodal data. Existing methods focus on capturing conversational relationships between speakers and short-distance context latent dependencies. Due to the limitation of GNN complexity, existing methods cannot capture long-distance contextual latent dependencies, which limits the performance of MER. In this paper, we propose a novel Efficient Long-distance Latent Relation-aware Graph Neural Network (ELR-GNN) for multi-modal emotion recognition, where conversational relationships between speakers, contextual semantic information, and long-range contextual latent dependencies of utterances can all be captured. Specifically, we first use pre-extracted text, video and audio features as input to Bi-LSTM to capture contextual semantic information and obtain low-level discourse features. We then use low-level discourse features to construct speaker graphs. To capture potential dependency information between distant contexts, we use a graph random neural network algorithm to randomly sample top-k nodes for information extraction. Furthermore, we combine early fusion and adaptive late fusion mechanisms to fuse latent dependency information between speaker relationship information and context. Finally, we obtain high-level discourse features and feed them into MLP for emotion prediction. In addition, there is currently no federated learning (FL) benchmark for MER. To promote the research of multimodal FL, we introduce FedMultimodal to implement an end-to-end modeling framework from data partitioning and feature extraction to FL benchmark algorithms and model evaluation. Extensive experiments have proven the superiority of the ELR-GNN architecture on two benchmark data sets.
Keywords:
Multimodal Emotion Recognition
Graph Neural Network
Long-distance Latent Dependencies
Federated Learning
Speaker Relationships
Journal
IF:
4
Papers:
3.8K
Citations:
4.8K
Organization
Cited Papers