.png){kind=icon}
.png){kind=icon}
Return
GraphVSum:graph guided multimodal video summarization
.png){kind=icon}
PRE
AI
DOI：10.1007/s00530-025-02084-x.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Multimodal video summarization is crucial for reducing redundancy and highlighting key moments in long or complex videos. However, current methods face challenges in integrating multimodal information deeply and handling their semantic relations. To address these issues, we propose GraphVSum, a novel framework that converts multimodal video summarization into a heterogeneous graph processing task. It constructs a heterogeneous graph of video frames and associated sentences, and optimizes their connections through intra-modal and inter-modal constraints. In GraphVSum, a dual fusion technique is proposed for multimodal embedding, combining shallow fusion with global attention and deep fusion via subgraph propagation. For generating accurate and contextually rich summaries, we propose a novel multimodal joint learning in GraphVSum, which includes a classification loss, an inter-modal coherence loss and an intra-modal diversity loss. Extensive experiments on the BLiSS, PlotSnap, Daily Mail and TVSum datasets demonstrate the effectiveness of GraphVSum in several metrics. The code of GraphVSum is publicly available at https://github.com/eMoLii/GraphVSum .
Keywords:
Video summarization
Heterogeneous graph
Multimodal fusion
Multimodal alignment
Journal
IF:
3.1
Papers:
2.7K
Citations:
2.7K
Organization
No organization information available