.png){kind=icon}
.png){kind=icon}
Return
A Multi-Modal Transformer network for action detection
.png){kind=icon}
OA
AI
DOI：10.1016/j.patcog.2023.109713.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
This paper proposes a novel multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our transformer network utilizes a new multi-modal attention mechanism that computes the correlations between different spatial and motion modalities combinations. Exploring such correlations for actions has not been attempted previously. To use the motion and spatial modality more effectively, we suggest an algorithm that corrects the motion distortion caused by camera move-ment. Such motion distortion, common in untrimmed videos, severely reduces the expressive power of motion features such as optical flow fields. Our proposed algorithm outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and ActivityNet. We also conducted comparative ex-periments on our new instructional activity dataset, including a large set of challenging classroom videos captured from elementary schools.Published by Elsevier Ltd.
Keywords:
Action detection
Transformer network
Optical flow
Motion features
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.