.png){kind=icon}
Return
Spatial-Temporal Side Adaptation with Difference-based Frame Selector for video class-incremental learning☆
.png){kind=icon}
PRE
AI
DOI：10.1016/j.displa.2026.103578.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Video class-incremental learning aims to continuously acquire new action categories while mitigating catastrophic forgetting. However, current methods typically suffer from excessive parameter overheads and fail to retain critical motion information within stored frames. To overcome these limitations, we propose a highly efficient framework driven by two novel components: a lightweight Spatial-Temporal Side Adaptation (STSA) module and a Difference-based Frame Selector (DFS). Instead of heavy fine-tuning, STSA decouples intermediate features from a CLIP-based ViT into global semantic and local spatial representations for independent temporal and spatial modeling. It subsequently fuses them and applies a global-local temporal orthogonal loss to explicitly capture complex temporal dynamics. To optimize replay efficiency under strict memory budgets, DFS leverages global-local temporal differences and inter-task representation drift to quantify memory-aware motion salience, and constructs a cumulative salience curve to adaptively sample the most discriminative frames. Extensive experiments on the UCF101, HMDB51, and UESTC-MMEA-CL datasets demonstrate that our approach significantly outperforms the state-of-the-art VCIL method under the same memory budgets, requiring only 42.55% of its trainable parameters.
Keywords:
Video class-incremental learning
Action recognition
Catastrophic forgetting
Spatial-temporal modeling
Journal
IF:
3.4
Papers:
2.3K
Citations:
3.2K
Organization
Cited Papers
No cited papers available