Joints-Centered Spatial-Temporal Features Fused Skeleton Convolution Network for Action Recognition

Wenfeng Song

doi:10.1109/TMM.2023.3324835

Joints-Centered Spatial-Temporal Features Fused Skeleton Convolution Network for Action Recognition

2024-01-01

2

PRE

AI

DOI：10.1109/TMM.2023.3324835

Original

Original request for help

Save

Abstract

En 中文

Skeleton-based action recognition is crucial for natural human-computer interaction, dynamic behavior analysis, and behavior surveillance. The key challenge is to effectively capture the intrinsic local-global clues of the activity. However, it remains challenging to efficiently leverage multidimensional information related to joints' local visual appearances, global spatial relationships, and coherent temporal cues. To address this challenge, we propose a joints-centered spatial-temporal feature-fused framework for action recognition, which exploits skeleton-based graph diffusion and convolution. Specifically, we employ Partial Differential Equation (PDE) based skeleton graph diffusion to automatically activate and diffuse the salient appearance features of joints. This approach simultaneously integrates the joints' appearance clues and their hierarchical relationships at both the super-pixel level and structure level. The diffused appearance-related features of the joints are further fused with skeleton-related spatial-temporal features, and the resulting fused features are fed into a skeleton convolution network for action recognition. Our method was extensively evaluated on two public datasets (NTU-RGBD and UWA3D), and the results demonstrate the improved accuracy and effectiveness of our approach. Our code will be public.

Keywords:

Skeleton

Feature extraction

Convolution

Visualization

Task analysis

Joints

Data mining

Skeleton-based action recognition

spatial-temporal feature fusion

PDE diffusion

Joints-Centered Spatial-Temporal Features Fused Skeleton Convolution Network for Action Recognition

Abstract

Journal

Organization