TEST: Temporal-spatial separated transformer for temporal action localization

Herun Wan

doi:10.1016/j.neucom.2024.128688

TEST: Temporal-spatial separated transformer for temporal action localization

2025-01-01

0

PRE

AI

DOI：10.1016/j.neucom.2024.128688

Original

Original request for help

Save

Abstract

En 中文

Temporal action localization is a fundamental task in video understanding. Existing methods fall into three categories: anchor-based, actionness-guided, and anchor-free. Anchor-based and actionness-guided models need huge computation resources to process redundant proposals or enumerate every possible proposal. Anchor- free models with lighter parameters become a more attractive option as temporal actions become more complex. However, they typically struggle to achieve high performance due to the need to aggregate global temporal-spatial features at every time step. To overcome this limitation, we design three efficient transformer- based architectures, bringing two advantages: (i) the global receptive field of transformers enables models to aggregate spatial and temporal at each time step, and (ii) the transformers could capture the moment-level feature, enhancing localization performance. Our designed architectures are adapted to any framework, thus we propose a simple but effective anchor-free framework named TEST. Compared to strong baselines, TEST achieves 0.96% to 3.20% improvement on two real-world datasets. Meanwhile, it improves time efficiency by 1.36 times and space efficiency by 1.08 times. Further experiments prove the effectiveness of TEST's modules. Implementation of our work is available at https://github.com/whr000001/TeST.

Keywords:

Video transformer

Temporal action localization

High efficiency

TEST: Temporal-spatial separated transformer for temporal action localization

Abstract

Journal

Organization