.png){kind=icon}
返回
M3S: Scene Graph Driven Multi-Granularity Multi-Task Learning for Multi-Modal NER
.png){kind=icon}
PRE
AI
DOI：10.1109/TASLP.2022.3221017.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Multi-modal Named Entity Recognition (MNER), which mainly focuses on enhancing text-only NER with visual information, has recently attracted considerable attention. Most current MNER models have made significant progress by jointly understanding visual and language modalities through layers of cross-modality attention. However, these approaches largely ignore the visual bias brought by the image contents and barely consider exploiting the multi-granularity representations and the interactions between visual objects, which are essential in recognizing ambiguous entities. In this paper, we propose a Scene graph driven Multi-modal Multi-granularity Multi-task learning (M3S) framework to better exploit visual and textual information in MNER. Specifically, to explicitly alleviate visual bias, we present a novel multi-task approach by employing the task of Named Entity Segmentation (NES) cascade with Named Entity Categorization (NEC). To obtain detailed visual semantics by explicitly modeling objects and relationships between paired objects, we construct scene graphs as a structured representation of the visual contents. Furthermore, a well-designed Multi-granularity Gated Aggregation (MGA) mechanism is introduced to capture inter-modality interactions and extract critical features for named entity recognition. Extensive experiments on two real public datasets demonstrate the effectiveness of our proposed M3S.
Keyword:
Visualization
Task analysis
Semantics
Feature extraction
Multitasking
Social networking (online)
Image segmentation
Named entity recognition
multi-modal learning
scene graph
期刊
I
IF:
5.1
论文数:
2.6K
被引数:
1.1W
机构
引用论文
分享收藏Improved analytical calculation model of spur gear mesh excitations with tooth profile deviations
PREAI
分享收藏Word-Character Graph Convolution Network for Chinese Named Entity Recognition用于中文命名实体识别的字-字图卷积网络
PREAI
分享收藏Cobalt and Copper Composite Oxides as Efficient Catalysts for Preferential Oxidation of CO in H2-Rich Stream钴和铜复合氧化物作为H2-Rich流中CO优先氧化的有效催化剂
PREAI
分享收藏Shifts in the ESR Spectra of Alkali‐Metal Atoms (Li, Na, K, Rb) on Helium Nanodroplets.png)
ChemPhysChem
OAAI
ChemPhysChemIF0
OAAI分享收藏Transposase-Derived Transcription Factors Regulate Light Signaling in
Arabidopsis转座酶衍生的转录因子调节光信号
拟南芥 ScienceOAAI
ScienceIF0
OAAI分享收藏Fermentation enzymes of Giardia intestinalis, pyruvate:ferredoxin oxidoreductase and hydrogenase, do not localize to its mitosomesMicrobiology
OAAI
MicrobiologyIF0
OAAI分享收藏
查看更多内容