Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning

Cong Yang

doi:10.1109/TGRS.2024.3359316

2024-01-01

Cong Yang

Zuchao Li

Lefei Zhang *

DOI：10.1109/TGRS.2024.3359316

原文链接

原文求助

摘要

En 中文

Recently, remote sensing image captioning (RSIC) has gained significant attention in the remote sensing community. Due to the significant differences in spatial resolution of remote sensing images, existing methods in this field have predominantly concentrated on the fine-grained extraction of remote sensing image features, but they cannot effectively handle the semantic consistency between visual features and textual features. To efficiently align the image-text, we propose a novel two-stage vision-language pre-training (VLP)-based approach to bootstrap interactive image-text alignment for RSIC, called BITA, which relies on the design of a lightweight interactive Fourier transformer (IFT) to better align remote sensing image-text features. The Fourier layer in the IFT is capable of extracting multiscale features of remote sensing images in the frequency domain, thereby reducing the redundancy of remote sensing visual features. Specifically, the first stage involves preliminary alignment through image-text contrastive learning (ITC), which aligns the learned multiscale remote sensing features from the IFT with textual features. In the second stage, the IFT connects the frozen image encoder with a large language model (LLM). Then, prefix causal language modeling (PCLM) is utilized to guide the text generation process using visual features. Ultimately, across the UCM-caption, RSICD, and NWPU-caption datasets, the experimental results clearly demonstrate that BITA outperforms other advanced comparative approaches.

Keyword:

Remote sensing

Visualization

Feature extraction

Transformers

Task analysis

Semantics

Discrete Fourier transforms

Fourier transformer

multimodal information alignment

remote sensing image captioning (RSIC)

vision-language pre-training (VLP)