.png){kind=icon}
返回
Parallel Pathway Dense Video Captioning With Deformable Transformer
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2022.3228821.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Dense video captioning is a very challenging task because it requires a high-level understanding of the video story, as well as pinpointing details such as objects and motions for a consistent and fluent description of the video. Many existing solutions divide this problem into two sub-tasks, event detection and captioning, and solve them sequentially ( localize-then-describe or reverse). Consequently, the final outcome is highly dependent on the performance of the preceding modules. In this paper, we decompose this sequential approach by proposing a parallel pathway dense video captioning framework that localizes and describes events simultaneously without any bottlenecks. We introduce a representation organization network at the branching point of the parallel pathway to organize the encoded video feature by considering the entire storyline. Then, an event localizer focuses to localize events without any event proposal generation network, a sentence generator describes events while considering the fluency and coherency of sentences. Our method has several advantages over existing work: (i) the final output does not depend on the output of the preceding modules, (ii) it improves existing parallel decoding methods by relieving the bottleneck of information. We evaluate the performance of PPVC on large-scale benchmark datasets, the ActivityNet Captions, and YouCook2. PPVC not only outperforms existing algorithms on the majority of metrics but also improves on both datasets by 5.4% and 4.9% compared to the state-of-the-art parallel decoding method.
Keyword:
Machine learning
deep learning
video and language
video captioning
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
Understanding Objects in Video: Object-Oriented Video Captioning via Structured Trajectory and Adversarial Learning.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAI分享收藏Advancement of reproductive activity, seasonal reduction in prolactin secretion and seasonal pelage changes in pubertal red deer hinds (Cervus elaphus) subjected to artificially shortened daily photoperiod or daily melatonin treatmentsReproductionOAAI
ReproductionIF0
OAAI分享收藏分享收藏The role of a critical left fronto-temporal network with its right-hemispheric homologue in syntactic learning based on word category information基于单词类别信息的关键左前-时间网络及其右半球同源物在句法学习中的作用
PREAI
分享收藏
查看更多内容