Deep Memory Network for Cross-Modal Retrieval

doi:10.1109/TMM.2018.2877122

Deep Memory Network for Cross-Modal Retrieval

2019-05-01

34

PRE

AI

DOI：10.1109/TMM.2018.2877122

Original

Original request for help

Save

Abstract

En 中文

With the explosive growth of multimedia data on the Internet, cross-modal retrieval has attracted a great deal of attention in both computer vision and multimedia communities. However, this task is challenging due to the heterogeneity gap between different modalities. Current approaches typically involve a common representation learning process that maps data from different modalities into a common space by linear or nonlinear embedding. Yet, most of them only handle the dual-modal situation and generalize poorly to complex cases that involve multiple modalities. In addition, they often require expensive fine-grained alignment of training data among diverse modalities. In this paper, we address these with a novel cross-modal memory network (CMMN), in which memory contents across modalities are simultaneously learned from end to end without the need of exact alignment. We further account for the diversity across multiple modalities using the strategy of adversarial learning. Extensive experimental results on several large-scale datasets demonstrate that the proposed CMMN approach achieves state-of-the-art performance in the task of cross-modal retrieval.

Keywords:

Cross-modal retrieval

memory network

deep learning

Deep Memory Network for Cross-Modal Retrieval

Abstract

AI Summary

Journal

Organization