A self-modified hypergraph neural network for multimodal relation extraction

Mingying Xu

doi:10.1016/j.ipm.2026.104673

A self-modified hypergraph neural network for multimodal relation extraction

2026-02-17

0

PRE

AI

DOI：10.1016/j.ipm.2026.104673

Original

Original request for help

Save

Abstract

En 中文

Multimodal Relation Extraction (MRE) aims to identify semantic relationships between entities in text-image pairs, playing a crucial role in multimodal understanding. Current graph-based MRE methods have achieved considerable success in modeling fine-grained multimodal interaction. However, there are limited by binary edges that require multi-hop reasoning and are prone to noise propagation. Moreover, these methods often overlook the exploitation of edge features, restricting structural optimization. To overcome these issues, we propose a Self-Modified Hypergraph Neural Network (SMHGNN). We construct a multimodal hypergraph where a hyperedge connects multiple nodes, enabling direct modeling of high-order semantic interactions. A key innovation is our mutually enhancing optimization mechanism between node features and hypergraph structure: hyperedges serve as feature carriers with attentional weighting, and the model iteratively self-modifies its connections through stacked HGNN layers to reduce noise and refine representations. Extensive experiments on public benchmarks show that SMHGNN achieves state-of-the-art performance, with a significant +1.84% F1 gain over the strongest baseline and a +3.13% improvement compared to the best graph-based method. The model also demonstrates strong robustness in low-resource scenarios.

Keywords:

Multimodal Relation Extraction

Hypergraph Neural Network

Semantic Relationships

Feature Carriers

Self-modification

A self-modified hypergraph neural network for multimodal relation extraction

Abstract

Journal

Organization