Exploring Transformer and Multilabel Classification for Remote Sensing Image Captioning

doi:10.1109/LGRS.2022.3198234

2022-01-01

Hitesh Kandala

Sudipan Saha

Biplab Banerjee

朱潇 (Xiao Xiang Zhu) *

DOI：10.1109/LGRS.2022.3198234

原文链接

原文求助

摘要

En 中文

High-resolution remote sensing images are now available with the progress of remote sensing technology. With respect to popular remote sensing tasks, such as scene classification, image captioning provides comprehensible information about such images by summarizing the image content in human-readable text. Most existing remote sensing image captioning methods are based on deep learning-based encoder-decoder frameworks, using convolutional neural network or recurrent neural network as the backbone of such frameworks. Such frameworks show a limited capability to analyze sequential data and cope with the lack of captioned remote sensing training images. Recently introduced Transformer architecture exploits self-attention to obtain superior performance for sequence-analysis tasks. Inspired by this, in this work, we employ a Transformer as an encoder-decoder for remote sensing image captioning. Moreover, to deal with the limited training data, an auxiliary decoder is used that further helps the encoder in the training process. The auxiliary decoder is trained for multilabel scene classification due to its conceptual similarity to image captioning and capability of highlighting semantic classes. To the best of our knowledge, this is the first work exploiting multilabel classification to improve remote sensing image captioning. Experimental results on the University of California (UC)-Merced caption dataset show the efficacy of the proposed method. The implementation details can be found in https://gitlab.lrz.de/ai4eo/captioningMultilabel.

Keyword:

Task analysis

Transformers

Remote sensing

Decoding

Feature extraction

Semantics

Training

Auxiliary task

image captioning

multitask learning

remote sensing