.png){kind=icon}
.png){kind=icon}
Return
Cross-multi-modal seamless training for image captioning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2025.130653.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Most CNN-LSTM and CNN-Transformer models for image captioning use Faster R-CNN to extract region features, resulting in a two-stage process that reduces efficiency and flexibility. To address these limitations, we propose a Cross Multi-modal Seamless Transformer (CMS-T) model, which uses the Swin Transformer for feature extraction and introduces a novel layered attention-driven architecture for refinement and decoding. Our model directly processes image grid features from the pre-trained Swin Transformer, seamlessly passing them to the refiner and decoder for caption generation. The proposed attention mechanism utilizes a cross-variance similarity score combined with bilinear pooling to distribute attention spatially and across channels. This approach effectively captures higher-order feature interactions, significantly improving computation time while maintaining the model size and enhancing caption quality. To validate the robustness of our approach, we also apply the same setup with an LSTM decoder, referred to as CMS-L, which incorporates a similar refiner module as the Transformer. Both cross-entropy and reinforcement learning techniques are employed. Numerous experiments demonstrate the effectiveness of this method for generating semantically rich image captions, as validated on the MSCOCO benchmark dataset. The results demonstrate the effectiveness of both inter- and intra-modal fusion strategies when integrated with Transformer-based architectures, surpassing conventional Faster R-CNN-based approaches. This integrated method significantly enhances fine-grained image captioning performance, achieving notable metrics such as a BLEU-4 score of 40.6 and a ROUGE-L score of 59.9. The code and corresponding results will be made available at https://github.com/shamimsareem/CMST-CMSL .
Journal
IF:
7.5
Papers:
2.9W
Citations:
10.2W