.png){kind=icon}
返回
A Dual-Feature-Based Adaptive Shared Transformer Network for Image Captioning
.png){kind=icon}
PRE
AI
DOI：10.1109/TIM.2024.3353830.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Current models exhibit notable efficacy in image-captioning tasks. Mainstream research shows that combining dual visual features enhances visual representations and brings a performance boost. However, the incorporation of dual visual features complicates computation and expands parameters, hindering streamlined model deployment. The selection of region features requires a pretrained object detector, neglecting the model's ease of use for new scenarios and data. In this article, we propose a dual-feature adaptive shared transformer network, capitalizing on the merits of grid and shallow patch features, while circumventing the extra complexity from dual channels. Specifically, we eschew complex features such as region features to facilitate straightforward dataset compilation and expedite inference. We propose an adaptive shared transformer block (AST) to conserve parameters and diminish the model's FLOPs. A gating mechanism is employed to adaptively compute the importance of each feature, thereby obtaining stronger visual features. Since using flattening grid features before a transformer often leads to a loss of crucial spatial information, we incorporate the learning of relative geometric information based on grid features into our proposed method. Our analysis of various feature fusion techniques reveals that the AST approach outperforms its counterparts in terms of FLOPs and model size while still achieving high performance. Extensive experiments on different datasets indicate that our model demonstrates competitive performance on MSCOCO and outperforms state-of-the-art models on small-scale datasets.
Keyword:
Deep learning
image captioning
transformer
期刊
IF:
5.9
论文数:
2.0W
被引数:
5.8W
机构
引用论文
Dual-Aspect Self-Attention Based on Transformer for Remaining Useful Life Prediction基于双方面自注意的变压器剩余寿命预测
OAAI
.png){kind=icon}
OAAI分享收藏分享收藏
分享收藏
查看更多内容