MPP-net: Multi-perspective perception network for dense video captioning

Yiwei Wei

doi:10.1016/j.neucom.2023.126523

MPP-net: Multi-perspective perception network for dense video captioning

2023-10-01

4

PRE

AI

DOI：10.1016/j.neucom.2023.126523

原文链接

原文求助

摘要

En 中文

Applying deformable transformer for dense video captioning has achieved great success recently. However, deformable transformer only explores local-perspective perception by attending to a small set of key sampling points, which will make the decoder short-sighted and generate semantically incoherent and contradictory dense captions for a long video. In this paper, we propose a novel Multi-Perspective Perception Network to improve this problem. We first introduce a hierarchical temporal-spatial summary method to generate global-perspective summary context for each decoder layer and avoid redundant information. Then our new designed multiperspective attention encourages the model to selectively incorporate the multi-perspective perception feature. Finally, we propose a novel multi-perspective generator to perform both multi-perspective feature fusion and caption generation. Experiments show that our proposed model outperforms previously published methods and achieves a competitive performance on ActivityNet Captions and YouCook2. The design of our model also shows the universality of other visual tasks that we obtain comparable results by applying our model for Object Detection and Paragraph Video Captioning.

Keyword:

Dense video captioning

Deformable transformer

Object detection

Paragraph video captioning

MPP-net: Multi-perspective perception network for dense video captioning

摘要

期刊

机构

引用论文