.png){kind=icon}
返回
Decoding before aligning: Scale-Adaptive Early-Decoding Transformer for visual grounding
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2025.129756.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Visual grounding, the task of precisely localizing objects within an image as described by natural language expressions, has recently seen significant advancements through the integration of transformer decoders and early-fusion mechanisms. These approaches aim to enhance the decoding of crucial grounding-related semantics and the generation of discriminative representations. However, they display significant limitations, including reduced generalizability due to the need for custom plug-ins, a gap in effectively bridging decoded visual and linguistic features, and difficulties in processing single-scale features within complex, hierarchical scenes. To address these issues, this study introduces the Scale-Adaptive Early-Decoding Transformer (SA-EDTR), a novel approach that combines decoding and early-fusion mechanisms within a single transformer layer. By positioning the decoding stage before the alignment stage, the decoder adaptively generates richer, context- aware representations that guide the aligning module. This design allows SA-EDTR to effectively overcome previous limitations in feature alignment. Additionally, we integrate a Scale-Adaptive Fusion network (SAF) to fuse multi-scale and heterogeneous representations from parallel Early-Decoding Transformer layers, enhancing the capture of spatial, coarse, and fine-grained semantic details. Extensive experiments demonstrate that SAEDTR outperforms current state-of-the-art methods in both REC and RES tasks across various mainstream visual grounding datasets, including RefCOCO, RefCOCO+, RefCOCOg-umd, and RefCOCOg-google. The results highlight the architecture's remarkable capability inaccurately localizing objects and interpreting intricate scenes.
Keyword:
Referring expression comprehension
Referring expression segmentation
Visual grounding
Vision-language
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏Cross-modality synergy network for referring expression comprehension and segmentation用于指称表达理解和分割的跨模态协同网络.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏
分享收藏
查看更多内容