.png){kind=icon}
返回
Unsupervised Modality-Transferable Video Highlight Detection With Representation Activation Sequence Learning
.png){kind=icon}
OA
AI
DOI：10.1109/TIP.2024.3372469.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Identifying highlight moments of raw video materials is crucial for improving the efficiency of editing videos that are pervasive on internet platforms. However, the extensive work of manually labeling footage has created obstacles to applying supervised methods to videos of unseen categories. The absence of an audio modality that contains valuable cues for highlight detection in many videos also makes it difficult to use multimodal strategies. In this paper, we propose a novel model with cross-modal perception for unsupervised highlight detection. The proposed model learns representations with visual-audio level semantics from image-audio pair data via a self-reconstruction task. To achieve unsupervised highlight detection, we investigate the latent representations of the network and propose the representation activation sequence learning (RASL) module with k-point contrastive learning to learn significant representation activations. To connect the visual modality with the audio modality, we use the symmetric contrastive learning (SCL) module to learn the paired visual and audio representations. Furthermore, an auxiliary task of masked feature vector sequence (FVS) reconstruction is simultaneously conducted during pretraining for representation enhancement. During inference, the cross-modal pretrained model can generate representations with paired visual-audio semantics given only the visual modality. The RASL module is used to output the highlight scores. The experimental results show that the proposed framework achieves superior performance compared to other state-of-the-art approaches.
Keyword:
Unsupervised
highlight detection
multimodal
representation activation sequence
期刊
IF:
13.7
论文数:
1.0W
被引数:
8.4W
机构
引用论文
Solution structure of the Lewis x oligosaccharide determined by NMR spectroscopy and molecular dynamics simulations.png)
Biochemistry
BiochemistryIF0
PREAI
分享收藏Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏分享收藏Viability of Hydrogen Water Chemistry for Protecting In-Vessel Components of Boiling Water ReactorsCORROSION
CORROSIONIF0
PREAI
分享收藏
查看更多内容