STFormer: Spatial-Temporal-Aware Transformer for Video Instance Segmentation

doi:10.1109/TNNLS.2024.3455551

STFormer: Spatial-Temporal-Aware Transformer for Video Instance Segmentation

2024-01-01

0

PRE

AI

Xinwang Liu (Xinwang Liu)

李肯立 (Kenli Li)

DOI：10.1109/TNNLS.2024.3455551

Original

Original request for help

Save

Abstract

En 中文

Video instance segmentation (VIS) is a challenging task, requiring handling object classification, segmentation, and tracking in videos. Existing Transformer-based VIS approaches have shown remarkable success, combining encoded features and instance queries as decoder inputs. However, their decoder inputs are low-resolution due to computational cost, resulting in a loss of fine-grained information, sensitivity to background interference, and poor handling of small objects. Moreover, the queries are randomly initialized without location information, hindering convergence efficiency and accurate object instance localization. To address these issues, we propose a novel VIS approach, STFormer, with a spatial-temporal feature aggregation (STFA) module and spatial-temporal-aware Transformer (STT). Specifically, STFA obtains robust high-resolution masked features efficiently for the decoder, while STT's location-guided instance query (LGIQ) improves initial instance queries. STFormer preserves more fine-grained information, improves convergence efficiency, and localizes object instance features accurately. Extensive experiments on YouTube-VIS 2019, YouTube-VIS 2021, and OVIS datasets show that STFormer outperforms mainstream VIS methods.

Keywords:

Transformers

Instance segmentation

Decoding

Interference

Head

Computer vision

Computational modeling

Motion segmentation

Convergence

Object recognition

Fine-grained information

object tracking

transformer

video instance segmentation (VIS)

STFormer: Spatial-Temporal-Aware Transformer for Video Instance Segmentation

Abstract

Journal

Organization

Cited Papers