Dynamic semantic prototype perception for text-video retrieval

doi:10.1016/j.imavis.2025.105515

Return

Dynamic semantic prototype perception for text-video retrieval

2025-05-01

PRE

Henghao Zhao

Rui Yan

Zechao Li *

DOI：10.1016/j.imavis.2025.105515

Original

Original request for help

Save

Abstract

En 中文

Semantic alignment between local visual regions and textual description is a promising solution for finegrained text-video retrieval task. However, existing methods rely on the additional object detector as the explicit supervision, which is unfriendly to real application. To this end, a novel Dynamic Semantic Prototype Perception (DSP Perception) is proposed that automatically learns, constructs and infers the dynamic spatiotemporal dependencies between visual regions and text words without any explicit supervision. Specifically, DSP Perception consists of three components: the spatial semantic parsing module, the spatio-temporal semantic correlation module and the cross-modal semantic prototype alignment. The spatial semantic parsing module is leveraged to quantize visual patches to reduce the visual diversity, which helps to subsequently aggregate the similar semantic regions. The spatio-temporal semantic correlation module is introduced to learn dynamic information between adjacent frames and aggregate local features belonging to the same semantic in the video as tube. In addition, a novel global-to-local alignment strategy is proposed for the cross-modal semantic prototype alignment, which provides spatio-temporal cues for cross-modal perception of dynamic semantic prototypes. Thus, the proposed DSP Perception enables to capture local regions and their dynamic information within the video. Extensive experiments conducted on four widely-used datasets (MSR-VTT, MSVD, ActivityNet-Caption and DiDeMo) demonstrate the effectiveness of the proposed DSP Perception by comparison with several state-of-the-art methods.

Keywords:

Video retrieval

Global-local

Fine-grained

Dynamic semantic prototype

Video representation learning