Coupled-dynamic learning for vision and language: Exploring Interaction between different tasks

doi:10.1016/j.patcog.2021.107829

Return

Coupled-dynamic learning for vision and language: Exploring Interaction between different tasks

2021-05-01

PRE

DOI：10.1016/j.patcog.2021.107829

Original

Original request for help

Save

Abstract

En 中文

Intensive research interests have been paid for the vision and language communities. Especially, image captioning task aims to generate natural language descriptions from the image content. Oppositely, image synthesis task aims to generate realistic images from natural language descriptions. Moreover, both of them can achieve promising results by using Long Short-Term Memory (LSTM), which models the sequence dynamics at each time step as hidden state. Nevertheless, the research on dynamics is often limited in the individual task, while there is no progress exploring the mutual relationship between dynamics in different tasks. In this work, we present a novel coupled-dynamic formulation that can iteratively reduce the distance between task-dependent dynamics in the training process. To embed adverse information into individual network, we construct dual-loss architectures to interactively align dynamics. We evaluate the proposed framework on Flickr8k, Flickr30k and MSCOCO datasets. Experimental results show that our approach can boost dual tasks together and achieve competing performances against stateof-the-art methods. (c) 2021 Elsevier Ltd. All rights reserved.

Keywords:

Image captioning

Image synthesis

Coupled dynamics

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.