.png){kind=icon}
Return
Scale-aware Multi-head Attention with Explainability for Image Captioning
.png){kind=icon}
PRE
AI
DOI：10.1007/978-981-95-5696-0_7.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Attention mechanisms, and specifically Transformer models, are dominating the field of image captioning. Current mainstream methods typically use grid image features as input and leverage a multi head attention-based encoder-decoder framework to contextualize information. However, the encoder's self-attention mechanism only captures visual relationships between fixed-size grid features, failing to account for varying object scales, which impacts visual feature representation and caption quality. To solve this problem, we propose a novel Scale-aware Multi-head Attention (SMA) to fully exploit multi-scale visual features for image captioning via introducing a scale-extension algorithm to attention. In particular, each head in the multi-head attention integrates distinct high-scale features into the fixed low-scale grid features, enabling the capture of more diverse and richer information. Nevertheless, despite SMA's excellent performance, the visual-language generation task has also sparked a demand for model explainability. To more accurately explain the proposed model, we introduce a multi-scale visual-language explanation network (MVEN) to accumulate relevancy maps from input to output. The explainability relevancy not only aggregates information in a multi-scale context but also generates more interpretable attention heatmaps in both visual and textual-level.
Keywords:
image captioning
scale-aware multi-head attention
attention explainability
Journal
P
IF:
0
Papers:
25
Citations:
0
Organization
Cited Papers
ShareSave
ShareSaveWhat Does BERT Look at? An Analysis of BERT’s Attention.png)
Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
OAAI
Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLPIF0
OAAIShareSaveShareSave
View more