.png){kind=icon}
Return
Weakly Supervised Temporal Adjacent Network for Language Grounding
.png){kind=icon}
OA
AI
DOI：10.1109/TMM.2021.3096087.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Temporal language grounding (TLG) is a fundamental and challenging problem for vision and language understanding. Existing methods mainly focus on fully supervised setting with temporal boundary labels for training, which, however, suffers expensive cost of annotation. In this work, we are dedicated to weakly supervised TLG, where multiple description sentences are given to an untrimmed video without temporal boundary labels. In this task, it is critical to learn a strong cross-modal semantic alignment between sentence semantics and visual content. To this end, we introduce a novel weakly supervised temporal adjacent network (WSTAN) for temporal language grounding. Specifically, WSTAN learns cross-modal semantic alignment by exploiting temporal adjacent network in a multiple instance learning (MIL) paradigm, with a whole description paragraph as input. Moreover, we integrate a complementary branch into the framework, which explicitly refines the predictions with pseudo supervision from the MIL stage. An additional self-discriminating loss is devised on both the MIL branch and the complementary branch, aiming to enhance semantic discrimination by self-supervising. Extensive experiments are conducted on three widely used benchmark datasets, i.e., ActivityNet-Captions, Charades-STA, and DiDeMo, and the results demonstrate the effectiveness of our approach.
Keywords:
Grounding
Semantics
Feature extraction
Visualization
Task analysis
Annotations
Training
Temporal language grounding
weakly supervised learning
multi-model understanding
multiple instance learning
Journal
IF:
9.7
Papers:
4.6K
Citations:
2.4W
Organization
Cited Papers
.png){kind=icon}
ShareSave
ShareSave
View more