.png){kind=icon}
返回
Interpretable Multimodal Capsule Fusion
.png){kind=icon}
PRE
AI
DOI：10.1109/TASLP.2022.3178236.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
With the development of social networking platform, multimodal sentiment analysis has become increasingly prominent. Existing models focus on capturing intramodal and intermodal interactions to produce effective modality representations. However, they overlook the study of interpretability which reveals how modalities interact with each other and which modality contributes most to the final prediction. In this paper, we propose an interpretable model called Interpretable Multimodal Capsule Fusion (IMCF) which integrates routing mechanism of Capsule Network (CapsNet) and Long Short-Term Memory (LSTM) to produce refined modality representations and provide interpretation. By constructing features of different modalities into input sequence, we are able to obtain highly expressive representation of intermodal dynamics due to the strong ability of LSTM to produce representation of sequence. As routing mechanism is applied during modality fusion and prediction stages, the value of routing coefficient can reveal the contributions of different modalities or dynamics, which provides interpretation. Meanwhile, routing mechanism can iteratively adjust the information flows of different modalities, which makes the process of modality fusion more reasonable. The experimental results show that our model achieves competitive performance on two benchmark datasets with effective modality fusion by LSTM and interpretation provided by routing mechanism.
Keyword:
Routing
Transformers
Adaptation models
Visualization
Brain modeling
Feature extraction
Data models
Capsule network
interpretation
LSTM
modality fusion
multimodal sentiment analysis
期刊
I
IF:
5.1
论文数:
2.6K
被引数:
1.1W
机构
引用论文
Entity-Sensitive Attention and Fusion Network for Entity-Level Multimodal Sentiment Classification
OAAI
OAAI分享收藏4-Phenylbutyrate Attenuates the ER Stress Response and Cyclic AMP Accumulation in DYT1 Dystonia Cell Models.png)
PLoS ONE
OAAI
PLoS ONEIF0
OAAI分享收藏Analyzing Multimodal Sentiment Via Acoustic- and Visual-LSTM With Channel-Aware Temporal Convolution Network使用通道感知的时间卷积网络通过声学和视觉LSTM分析多模式情感
PREAI
分享收藏
查看更多内容