Multi-Branch Distance-Sensitive Self-Attention Network for Image Captioning

doi:10.1109/TMM.2022.3169061

Multi-Branch Distance-Sensitive Self-Attention Network for Image Captioning

2023-01-01

10

PRE

AI

DOI：10.1109/TMM.2022.3169061

Original

Original request for help

Save

Abstract

En 中文

Self-attention (SA) based networks have achieved great success in image captioning, constantly dominating the leaderboards of online benchmarks. However, existing SA networks still suffer from distance insensitivity and low-rank bottleneck. In this paper, we aim to optimize SA in terms of two aspects, thereby addressing the above issues. First, we introduce a Distance-sensitive Self-Attention (DSA), which considers the raw geometric distances between query-key pairs in the 2D images during SA modeling. Second, we present a simple yet effective approach, named Multi-branch Self-Attention (MSA) to compensate for the low-rank bottleneck. MSA treats a multi-head self-attention layer as a branch and duplicates it multiple times to increase the expressive power of SA. To validate the effectiveness of the two designs, we apply them to the standard self-attention network, and conduct extensive experiments on the highly competitive MS-COCO dataset. We achieve new state-of-the-art performance on both the local and online test sets, i.e., 135.1% CIDEr on the Karpathy split and 135.4% CIDEr on the official online split.

Keywords:

Transformers

Visualization

Head

Feature extraction

Encoding

Computer architecture

Computational modeling

Image captioning

multi-branch techniques

distance-sensitive positional embedding

Multi-Branch Distance-Sensitive Self-Attention Network for Image Captioning

Abstract

Journal

Organization