.png){kind=icon}
.png){kind=icon}
Return
Type-Specific Modality Alignment for Multi-Modal Information Extraction
.png){kind=icon}
PRE
AI
DOI：10.1109/LSP.2024.3396705.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Multi-modal information extraction aims to identify structured information, such as entities or relations between entities, from text with the help of visual clues. Although existing studies have achieved great progress, they mainly focused on modality interactions in the global space while neglecting fine-grained modality alignments under the semantic subspace specific to each entity type or relation type. To solve this problem, we propose a multi-space modality alignment method (MSMA) in this letter. The core of our model is a type-specific modality interaction module (TMI), which constructs a unique semantic subspace for each entity/relation type and independently performs type-specific modality alignments under each subspace. To enable mutual promotion between different types, a global modality integration module (GMI) is designed to learn the associations between different subspaces. Furthermore, we execute these two modules iteratively for high-level semantic fusion. Extensive experiments on three benchmark datasets show that our model significantly outperforms advanced methods.
Keywords:
Multi-modal information extraction
visual clue
type-specific modality alignment
global modality integration
Journal
IF:
9.6
Papers:
1.1W
Citations:
1.7W