.png){kind=icon}
返回
Semi-supervised cross-modal retrieval with graph-based semantic alignment network
.png){kind=icon}
PRE
AI
DOI：10.1016/j.compeleceng.2022.108218.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Semi-supervised cross-modal retrieval is an eclectic paradigm which learns common representa-tions via exploiting underlying semantic information from both labeled and unlabeled data. Most existing methods ignore the rich semantic information of text data and are unable to fully utilize the text data in common representation learning. Moreover, they only considered the correlation of the data with the same semantic label, but ignored the correlation between the data with different semantic label. In this paper, we propose a novel semi-supervised cross-modal retrieval method, called Graph-based Semantic Alignment Network (GSAN), which learns common representation by aligning the features of different modalities with semantic embeddings of text data. Firstly, we design a Deep Supervised Semantic Encoding (DSSE) module to train the semantic projector and label predictor which can exploit the semantic embeddings and the predicted labels from unlabeled data of text modality. Then, GAN-based Bidirectional Fusion (GBF) module is designed to learn the mapping networks of two modalities (image and text). In order to make the mapping networks generate semantically discriminative and modality -invariant representations, we utilize the underlying semantic information exploited by DSSE to construct Graph-based Triplet Constraint (GTC) which can enforce feature embeddings from the semantically-matched (image and text) pairs to be more similar and push those mismatched ones away. By the benefit of fully using of semantic information, our approach can only use fewer label data and achieves the performance of state-of-the-art methods. In addition, since we only utilize the mapping networks trained in GBF module to generate common representations in referring stage, our approach is efficient and time saving in real world application. Extensive experiments on four widely-used datasets show the effectiveness of GSAN.
Keyword:
Cross-modal retrieval
Semi-supervised learning
Deep neural network
Generative adversarial network
期刊
C
IF:
4.9
论文数:
6.7K
被引数:
1.3W
机构
暂无机构信息
引用论文
The effect of decomposition of beta-phase Zr-20 at% Nb on hydrogen partitioning with alpha-zirconium
PREAI
分享收藏Solution Structure of Monomeric Peptide YY Supports the Functional Significance of the PP-Fold,.png)
Biochemistry
BiochemistryIF0
PREAI
分享收藏The effect of a novel extracorporeal cytokine hemoadsorption device on IL-6 elimination in septic patients: A randomized controlled trialPLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical NetworkCCL: 分层网络多粒度融合的跨模态相关学习OAAI
OAAI分享收藏
查看更多内容