Multimodal graph neural network for video procedural captioning

Lei Ji

doi:10.1016/j.neucom.2022.02.062

Multimodal graph neural network for video procedural captioning

2022-06-01

6

PRE

AI

DOI：10.1016/j.neucom.2022.02.062

原文链接

原文求助

摘要

En 中文

Video procedural captioning aims to generate detailed descriptive captions for all steps in a long instructional video. The peculiarity of this problem is the procedural dependency between the events to generate consistent captions among the video. However, existing video (dense) captioning methods only consider intra-event or sequential inter-event context and are hard to model the non-sequential context dependency between events. In this paper, inspired by the recent success of graph neural networks in capturing the relations for structured data, we propose a novel Multimodal Graph Neural Network (MGNN) for dense video procedural captioning in capturing the procedural structure between events. Specifically, we construct temporal sequential graph and semantic non-sequential graph for a multi modal heterogeneous graph. Moreover, we adopt the graph neural network to enhance the visual and text features, and fuse both features for further caption generation. Extensive experiments demonstrate the proposed MGNN is effective in generating coherent captions on both the Youcook2 and Activitynet Captions benchmark.(c) 2022 Elsevier B.V. All rights reserved.

Keyword:

Multimodal video captioning

Graph neural network

Multimodal graph neural network for video procedural captioning

摘要

期刊

机构

引用论文