.png){kind=icon}
Return
Visual relation-aware and knowledge-guided multi-modal relation extraction
.png){kind=icon}
PRE
AI
DOI：10.1007/s10586-025-05731-0.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Multi-modal relation extraction (MRE) aims to identify semantic relations between entities in text by leveraging associated visual information. While existing methods have made progress, they face two key limitations: (1) overlooking fine-grained visual relationships between objects that provide critical contextual clues, and (2) failing to utilize external knowledge that could enhance understanding of entities and textual context. To address these challenges, we propose DGF-PT, a novel framework that integrates Visual Relations and Multi-grained Knowledge for MRE. Our approach first constructs a unified multi-modal graph by aligning visual objects and textual tokens through scene graphs, where diverse relations (e.g., spatial, action-based) are explicitly modeled. A relation-aware Transformer then propagates information across the graph while preserving modality-specific and cross-modal interactions. Additionally, we employ large language models (LLMs) to generate entity-level descriptions and sentence-level explanations, which are selectively fused via cross-attention to enrich textual representations. Experiments on the MNRE benchmark show that DGF-PT achieves state-of-the-art performance (85.26% F1), outperforming prior methods by 2.71%.
Keywords:
Multi-modal relation extraction
Visual relations
Scene graphs
Large language models
Journal
C
IF:
0
Papers:
691
Citations:
1
Organization
No organization information available
Cited Papers
No cited papers available