Geometric understanding in spatial contexts for image captioning

doi:10.1016/j.image.2026.117645

Geometric understanding in spatial contexts for image captioning

2026-07-01

3

PRE

AI

M

Mohammad Alamgir Hossain

DOI：10.1016/j.image.2026.117645

Original

Original request for help

Save

Abstract

En 中文

Accurate image captioning requires not only recognizing objects but also reasoning about their spatial layout. However, many captioning models remain largely appearance-driven and encode object geometry only weakly, which limits spatially grounded descriptions in multi-object scenes. We propose GeoL (Geometric LSTM), a geometry-aware captioning framework that injects continuous RoI bounding-box attributes (normalized coordinates, centroids, area, perimeter, aspect ratio, and compactness) directly into the attention computation. GeoL introduces two complementary modules: an encoder-side Geometric Attention Block (GeoL-GeoAtt) that refines region features via geometry-conditioned multi-head attention, and a decoder-side GeoL-Att block that performs language-conditioned attention over geometry-enhanced regions with lightweight spatial and channel reweighting. GeoL is trained with cross-entropy and further optimized with self-critical sequence training to directly maximize CIDEr. On MSCOCO (COCO 2014, Karpathy split), GeoL achieves 38.4 BLEU-4 and 120.8 CIDEr under cross-entropy training, and 39.9 BLEU-4 and 129.9 CIDEr after SCST fine-tuning, demonstrating competitive performance against strong attention and Transformer-era baselines while improving spatial coherence in complex scenes. Code is available at https://github.com/alamgirustc/GeoL.

Keywords:

Geometric feature

Geometric attention

Image captioning

GeoL-GeoAtt

GeoL-Att block

Encoder-decoder

Geometric understanding in spatial contexts for image captioning

Abstract

Journal

Organization

Cited Papers