Learning Complementary Spatial-Temporal Transformer for Video Salient Object Detection

doi:10.1109/TNNLS.2023.3243246

2024-08-01

PRE

DOI：10.1109/TNNLS.2023.3243246

原文链接

原文求助

摘要

En 中文

Besides combining appearance and motion information, another crucial factor for video salient object detection (VSOD) is to mine spatial-temporal (ST) knowledge, including complementary long-short temporal cues and global-local spatial context from neighboring frames. However, the existing methods only explored part of them and ignored their complementarity. In this article, we propose a novel complementary ST transformer (CoSTFormer) for VSOD, which has a short-global branch and a long-local branch to aggregate complementary ST contexts. The former integrates the global context from the neighboring two frames using dense pairwise attention, while the latter is designed to fuse long-term temporal information from more consecutive frames with local attention windows. In this way, we decompose the ST context into a short-global part and a long-local part and leverage the powerful transformer to model the context relationship and learn their complementarity. To solve the contradiction between local window attention and object motion, we propose a novel flow-guided window attention (FGWA) mechanism to align the attention windows with object and camera movements. Furthermore, we deploy CoSTFormer on fused appearance and motion features, thus enabling the effective combination of all three VSOD factors. Besides, we present a pseudo video generation method to synthesize sufficient video clips from static images for training ST saliency models. Extensive experiments have verified the effectiveness of our method and illustrated that we achieve new state-of-the-art results on several benchmark datasets.

Keyword:

Transformers

Feature extraction

Context modeling

Aggregates

Optical flow

Object detection

Computational modeling

Attention models

optical flow

saliency detection

transformer

video salient object detection (VSOD)

期刊

IEEE Transactions on Neural Networks and Learning Systems

IF:

8.9

论文数:

7.6K

被引数:

7.2W

机构

mohamed bin zayed university of artificial intelligence mbzuai

学者数:

330

论文数: 332

被引数: 0

Northwestern Polytechnical University

学者数:

4.6W

论文数: 3.7W

被引数: 5.3W

引用论文

Judgment Capacity, Fear of Falling, and the Risk of Falls in Community-Dwelling Older Adults: The Progetto Veneto Anziani Longitudinal Study 社区居住的老年人的判断能力，对跌倒的恐惧和跌倒的风险: Progetto Veneto Anziani纵向研究

Rejuvenation Research

IF0

2020-06-01

OAAI

Caterina Trevisan; Bruno M. Zanforlini; Stefania Maggi; Marianna Noale; Federica Limongi; Marina De Rui; Maria Chiara Corti; Egle Perissinotto; Anna-Karin Welmer; Enzo Manzato; Giuseppe Sergi

Grain Boundary Strengthening Mechanism of Tungsten Containing 9 to 12% Chromium Ferritic Heat Resistant Steels at 650.DEG.C.

Journal of the Society of Materials Science, Japan

IF0

2003-01-01