Rwkv-vg: visual grounding with RWKV-driven encoder-decoder framework

doi:10.1007/s00530-025-01720-w

Rwkv-vg: visual grounding with RWKV-driven encoder-decoder framework

2025-02-21

0

PRE

AI

DOI：10.1007/s00530-025-01720-w

Original

Original request for help

Save

Abstract

En 中文

Visual grounding is a fundamental task that bridges vision and language, aiming to accurately associate natural language queries with specific regions in an image. Existing approaches, predominantly based on Transformers or CNNs, struggle with balancing computational efficiency and fine-grained semantic alignment. In this paper, we propose RWKV-VG, the first visual grounding framework entirely built on the RWKV architecture. Leveraging RWKV's unique ability to combine RNN-like sequential modeling with Transformer-like attention, our model efficiently achieves both intra-modal and cross-modal reasoning. The framework consists of a RWKV-based visual encoder, a RWKV-based linguistic encoder, and a RWKV-based visual-linguistic decoder, complemented by a learnable [REG] token designed for box regression. Comprehensive evaluations on benchmark datasets, including ReferItGame and the RefCOCO series, demonstrate the superiority of RWKV-VG, achieving state-of-the-art performance with rapid convergence. Ablation studies further confirm the effectiveness of the RWKV modules and the [REG] token design. Our work establishes RWKV as a compelling alternative to conventional architectures for visual grounding tasks. To facilitate future research, the code and pre-trained models are released at https://github.com/nianfd/RWKV-VG.

Keywords:

Visual grounding

RWKV

Encoder-decoder

Cross-modal learning

Rwkv-vg: visual grounding with RWKV-driven encoder-decoder framework

Abstract

Journal

Organization