Parameter-Efficient Reparameterization Tuning for Remote Sensing Image-Text Retrieval

doi:10.1109/TGRS.2025.3558246

Parameter-Efficient Reparameterization Tuning for Remote Sensing Image-Text Retrieval

2025-01-01

0

PRE

AI

DOI：10.1109/TGRS.2025.3558246

原文链接

原文求助

摘要

En 中文

Vision language models have been gradually adapted to various tasks of remote sensing (RS) domain with full fine-tuning paradigm, e.g., RS image-text retrieval (RSITR). Superior performance enhancements have proven the powerful generalization and robustness of vision language models. However, full fine-tuning vision language models is resource-intensive and poses risks of overfitting. Moreover, existing RSITR methods usually assume that RS images correspond to text captions one by one and utilize the bidirectional matching (BiMa) training objective, which is not aligned with evaluation benchmarks and real-world applications. To tackle the mentioned problems, we propose a novel parameter-efficient reparameterization tuning with ranking and matching (PERT-RaMa) framework, which effectively migrates the vision language model [i.e., contrastive language-image pre-training (CLIP)] to RSITR task. To overcome the overfitting issue, we build a lightweight, plug-and-play module called Kronecker product for low-rank adaptation (KPLoRA). KPLoRA obtains a higher intrinsic rank with fewer parameters. Furthermore, we design the RaMa training method that converts RSITR task into one-to-one matching and one-to-many ranking, which is aligned with current RSITR benchmarks and accelerates training speed through removal of unessential computations. Comprehensive experiments on three public RSITR benchmarks demonstrate that the effectiveness and efficiency of the proposed retrieval model. Our method outperforms full fine-tuning methods without CLIP by nearly 5%-10%, and achieves comparable or superior retrieval capability than CLIP with full fine-tuning and parameter-efficient fine-tuning (PEFT). Furthermore, our RaMa training method increases the training speed by 5x compared to current training method.

Keyword:

Tuning

Training

Adaptation models

Remote sensing

Feature extraction

Matrix decomposition

Visualization

Overfitting

Benchmark testing

Transformers

Kronecker product

low-rank adaptation (LoRA)

parameter-efficient reparameterization tuning (PERT)

ranking and matching (RaMa)

remote sensing image-text retrieval (RSITR)

Parameter-Efficient Reparameterization Tuning for Remote Sensing Image-Text Retrieval

摘要

期刊

机构

引用论文