.png){kind=icon}
Return
Interactive Visual Cue Refinement for Multi-modal Named Entity Recognition
.png){kind=icon}
PRE
AI
DOI：10.1007/978-981-95-5640-3_31.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
With the continuous growth of multi-modal data on social media platforms, traditional Named Entity Recognition has rendered insufficient for handling contemporary data formats. Consequently, researchers proposed Multi-modal Named Entity Recognition (MNER). Existing studies focus on capturing the visual regions corresponding to entities to assist in entity recognition. However, these approaches still struggle to mitigate interference from visual regions that are irrelevant to the entities. To address this issue, we propose an innovative framework, Interactive Visual Cue Refinement(IVCR) for MNER, to accurately capture visual cues (object-level visual regions) associated with entities. We leverage prompts to represent the semantic information of entity categories, which helps us assess visual cues and minimize interference from those irrelevant to the entities. Furthermore, we designed an interaction transformer that operates in two stages, first within each modality and then between modalities-to refine visual cues by learning from a frozen visual encoder, thereby reducing differences between text and visual modalities. Comprehensive experiments were conducted on two public datasets, Twitter15 and Twitter17. The results and detailed analyses demonstrate that our method exhibits robust and competitive performance.
Keywords:
Multi-modal Named Entity Recognition
Multi-modal Interaction
Prompt Learning
Transformer
Journal
W
IF:
0
Papers:
32
Citations:
0
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSaveLearning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NER
OAAI
OAAIShareSave
View more