.png){kind=icon}
返回
Memory based fusion for multi-modal deep learning
.png){kind=icon}
OA
AI
DOI：10.1016/j.inffus.2020.10.005.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The use of multi-modal data for deep machine learning has shown promise when compared to uni-modal approaches with fusion of multi-modal features resulting in improved performance in several applications. However, most state-of-the-art methods use naive fusion which processes feature streams independently, ignoring possible long-term dependencies within the data during fusion. In this paper, we present a novel Memory based Attentive Fusion layer, which fuses modes by incorporating both the current features and longterm dependencies in the data, thus allowing the model to understand the relative importance of modes over time. We introduce an explicit memory block within the fusion layer which stores features containing longterm dependencies of the fused data. The feature inputs from uni-modal encoders are fused through attentive composition and transformation followed by naive fusion of the resultant memory derived features with layer inputs. Following state-of-the-art methods, we have evaluated the performance and the generalizability of the proposed fusion approach on two different datasets with different modalities. In our experiments, we replace the naive fusion layer in benchmark networks with our proposed layer to enable a fair comparison. Experimental results indicate that the MBAF layer can generalize across different modalities and networks to enhance fusion and improve performance.
Keyword:
Attention
Generalized multi-modal fusion
Memory networks
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
15.5
论文数:
4.2K
被引数:
2.7W
机构
暂无机构信息
引用论文
Reflex effects from Golgi tendon organ (Ib) afferents are unchanged after spinal cord lesions in humans.png)
Neurology
NeurologyIF0
PREAI
分享收藏DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity RecognitionIEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAI分享收藏Information fusion in visual question answering: A Survey视觉问答中的信息融合: 一项调查INFORMATION FUSION
INFORMATION FUSIONIF15.5
PREAI
分享收藏A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition面向数据驱动的情感识别的多模态信息融合快照研究与实现INFORMATION FUSION
INFORMATION FUSIONIF15.5
PREAI
分享收藏Emotion recognition using deep learning approach from audio-visual emotional big dataINFORMATION FUSION
INFORMATION FUSIONIF15.5
PREAI
分享收藏Attention-based fusion network for human eye-fixation prediction in 3D imagesOPTICS EXPRESS
OAAI
OPTICS EXPRESSIF3.3
OAAI分享收藏
查看更多内容