.png){kind=icon}
返回
Type-Specific Modality Alignment for Multi-Modal Information Extraction
.png){kind=icon}
PRE
AI
DOI：10.1109/LSP.2024.3396705.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Multi-modal information extraction aims to identify structured information, such as entities or relations between entities, from text with the help of visual clues. Although existing studies have achieved great progress, they mainly focused on modality interactions in the global space while neglecting fine-grained modality alignments under the semantic subspace specific to each entity type or relation type. To solve this problem, we propose a multi-space modality alignment method (MSMA) in this letter. The core of our model is a type-specific modality interaction module (TMI), which constructs a unique semantic subspace for each entity/relation type and independently performs type-specific modality alignments under each subspace. To enable mutual promotion between different types, a global modality integration module (GMI) is designed to learn the associations between different subspaces. Furthermore, we execute these two modules iteratively for high-level semantic fusion. Extensive experiments on three benchmark datasets show that our model significantly outperforms advanced methods.
Keyword:
Multi-modal information extraction
visual clue
type-specific modality alignment
global modality integration
期刊
IF:
9.6
论文数:
1.1W
被引数:
1.7W
机构
引用论文
.png){kind=icon}
分享收藏分享收藏
查看更多内容