.png){kind=icon}
Return
Retrieval Augmented Convolutional Encoder-decoder Networks for Video Captioning
.png){kind=icon}
PRE
AI
DOI：10.1145/3539225.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Video captioning has been an emerging research topic in computer vision, which aims to generate a natural sentence to correctly reflect the visual content of a video. The well-established way of doing so is to rely on encoder-decoder paradigm by learning to encode the input video and decode the variable-length output sentence in a sequence-to-sequence manner. Nevertheless, these approaches often fail to produce complex and descriptive sentences as natural as those from human being, since the models are incapable of memorizing all visual contents and syntactic structures in the human-annotated video-sentence pairs. In this article, we uniquely introduce a Retrieval Augmentation Mechanism (RAM) that enables the explicit reference to existing video-sentence pairs within any encoder-decoder captioning model. Specifically, for each query video, a video-sentence retrieval model is first utilized to fetch semantically relevant sentences from the training sentence pool, coupled with the corresponding training videos. RAM then writes the relevant video-sentence pairs intomemory and reads the memorized visual contents/syntactic structures in video-sentence pairs from memory to facilitate the word prediction at each timestep. Furthermore, we present Retrieval Augmented Convolutional Encoder-Decoder Network (R-ConvED), which novelly integrates RAM into convolutional encoder-decoder structure to boost video captioning. Extensive experiments on MSVD, MSR-VTT, Activity Net Captions, and VATEX datasets validate the superiority of our proposals and demonstrate quantitatively compelling results.
Keywords:
Video captioning
deep convolutional neural networks
Journal
IF:
6
Papers:
2.0K
Citations:
5.4K
Organization
Cited Papers
ShareSaveShareSaveShareSaveShareSaveShareSaveVideo captioning with boundary-aware hierarchical language decoding and joint video prediction.png)
NEUROCOMPUTING
OAAI
NEUROCOMPUTINGIF6.5
OAAIShareSaveEnhancing the alignment between target words and corresponding frames for video captioningPATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
ShareSave
View more