Attention-supervised transformer for grounded image captioning

Xiangxiang Zhang

doi:10.1007/s00530-026-02684-1

Return

Attention-supervised transformer for grounded image captioning

2026-09-27

PRE

DOI：10.1007/s00530-026-02684-1

Original

Original request for help

Save

Abstract

En 中文

Image captioning, which involves generating natural language captions for given images, represents a highly challenging task and has been applied in various fields such as medical report and news article generation. Although the incorporation of attention mechanisms has significantly enhanced the performance of image captioning models, the alignment between image regions and descriptive words often remains weakly constrained, leading to imprecise captions. In this paper, we propose a Transformer-based Part-of-Speech enhanced Stacked Cross Attention Network (POS-SCAN-T), designed to improve grounded image captioning by leveraging attention weights from the Attention-Supervised Module (ASM) to guide the decoding process. Prior to model training, we first construct the ASM by integrating region features extracted via the Visual Encoding Module (VEM) with textual features encoded by the Text Embedding Module (TEM), which learns alignment-oriented attention weights. Subsequently, the Refined Encoder Module (REM) is introduced to further refine the region features. During decoding, the learned attention weights are employed to supervise the alignment between the refined region features and part-of-speech-filtered textual features. Experiments on the Flickr30k Entities and MS COCO benchmarks demonstrate that the POS-SCAN-T performs competitively against contemporary image captioning methods on some metrics. Compared to the baseline, our model achieves relative improvements of 2.1% and 3.5% in CIDEr scores on the two datasets, respectively, along with a consistent gain of 1.2% in SPICE scores, thereby clearly affirming the effectiveness of the proposed enhancements.

Keywords:

Image captioning

Region-word alignment

Cross-attention

Similarity

Visual grounding