Generative External Knowledge for Zero-shot Action Recognition

doi:10.1016/j.eswa.2025.127420

Return

Generative External Knowledge for Zero-shot Action Recognition

2025-06-01

PRE

DOI：10.1016/j.eswa.2025.127420

Original

Original request for help

Save

Abstract

En 中文

Zero-Shot Action Recognition (ZSAR) aims to infer new action classes without any samples of those classes. Traditional methods of acquiring language and visual knowledge limit the relevance and richness of such knowledge, complicating the generalization to the recognition of unseen classes. To address this issue, this paper proposes a novel method based on generative external knowledge for ZSAR. Specifically, high-quality objects are generated to serve as language knowledge of video samples using vision-language large-scale models. Meanwhile, class images are generated from class descriptions using text-to-image large-scale models to serve as visual knowledge of classes. Subsequently, a CLIP pre-trained encoder, combined with a unified multi-stream Transformer framework, learns the enriched cross-modal representations through robust interaction between dual independent encoders. Later, this paper presents a novel predictive model based on Generative Knowledge and Multi Prototype CLIP (GK-MP-CLIP), which leverages the quantity of class images to form a multi-prototype representation for classification support. The proposed method fully applies generative visual and language knowledge to construct the relationship between videos and classes, thereby enhancing ZSAR performance. Experimental results demonstrate significant advantages of the proposed method in enhancing the accuracy of zero-shot recognition for object-interactive actions. This indicates that selecting knowledge sources suited to dataset characteristics can improve ZSAR performance.

Keywords:

Zero-shot action recognition

Generative external knowledge

Cross-modal representation