.png){kind=icon}
返回
Deep Multimodal Complementarity Learning
.png){kind=icon}
PRE
AI
DOI：10.1109/TNNLS.2022.3165180.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Complementarity plays a significant role in the synergistic effect created by different components of a complex data object. Complementarity learning on multimodal data has fundamental challenges of representation learning because the complementarity exists along with multiple modalities and one or multiple items of each modality. Also, an appropriate metric is needed for measuring the complementarity in the representation space. Existing methods that rely on similarity-based metrics cannot adequately capture the complementarity. In this work, we propose a novel deep architecture for systematically learning the complementarity of components from multimodal multi-item data. The proposed model consists of three major modules: 1) unimodal aggregation for extracting the intramodal complementarity; 2) cross-modal fusion for extracting the intermodal complementarity at the modality level; and 3) interactive aggregation for extracting the intermodal complementarity at the item level. To quantify complementarity, we utilize the TUBE distance metric to measure the difference between the composited data object and its label in the representation space. Experiments on three real datasets show that our model outperforms the state-of-the-art by +6.8% of mean reciprocal rank (MRR) on object classification and +3.0% of MRR on hold-out item prediction. Qualitative analyses reveal that complementarity is significantly different from similarity.
Keyword:
Electron tubes
Measurement
Data models
Task analysis
Learning systems
Visualization
Representation learning
Complementarity modeling
deep learning
multimodal machine learning
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
Strength and permeability potentials of cement-modified desert sand for roads construction purpose用于道路建设的水泥改性沙漠砂的强度和渗透潜力
PREAI
分享收藏Coupling of Redox Conditions of Mantle Melting and Copper and Sulfur Contents in Primary Magmas of the Tolbachinsky Dol (Kamchatka) and Juan de Fuca Ridge (Pacific Ocean).png)
Petrology
PetrologyIF0
PREAI
分享收藏
查看更多内容