Dynamic interaction networks for image-text multimodal learning

Wenshan Wang

doi:10.1016/j.neucom.2019.10.103

Dynamic interaction networks for image-text multimodal learning

2020-02-01

3

PRE

AI

DOI：10.1016/j.neucom.2019.10.103

原文链接

原文求助

摘要

En 中文

Recently, there is a surge of interest in image-text multimodal representation learning, and many neural network based models have been proposed aiming to capture the interaction between two modalities with different forms of functions. Despite their success, a potential limitation of these methods is insufficient to model all kinds of interactions with a set of static parameters. To alleviate this problem, we present a dynamic interaction network, in which the parameters of the interaction function are dynamically generated by a meta network. Additionally, to provide necessary multimodal features that the meta network needs, we propose a new neural module called Multimodal Transformer. Experimentally, we not only make a comprehensively quantitative evaluation on four image-text tasks, but also show some interpretable analyses of our models, revealing the internal working mechanism of the dynamic parameter learning. (C) 2019 Elsevier B.V. All rights reserved.

Keyword:

Multimodal learning

Dynamic parameters prediction

Deep neural networks

Dynamic interaction networks for image-text multimodal learning

摘要

AI总结

期刊

机构

引用论文