Named entity recognition and coreference resolution using prompt-based generative multimodal

doi:10.1007/s40747-025-02122-1

Return

Named entity recognition and coreference resolution using prompt-based generative multimodal

2025-11-08

DOI：10.1007/s40747-025-02122-1

Original

Save

View PDF

Abstract

En 中文

Named Entity Recognition (NER) and Coreference Resolution (CR) are both of pivotal significance in the domain of natural language processing. The accurate detection of pronouns and noun phrases has been shown to have a direct impact on cross-sentence and even cross-modal semantic understanding. However, existing methods often suffer from inadequate semantic fusion in multimodal scenarios, leading to issues such as text-image mismatches and incorrect referential resolution. The present paper proposes a multimodal generation framework that integrates object detection with prompt-based strategies. First, a prompt mechanism is employed to embed entity types into the visual channel using object detection results and hierarchical features extracted by a visual Transformer. This design enhances multimodal NER. Then, the PGMCR model is constructed, which aligns the pronouns and noun phrases identified by NER with the image regions based on prompt information to achieve generative multimodal co-reference resolution. The experimental findings on the public CIN dataset demonstrate that PGMNER attains an F1 score of 96.55% in the NER task, which is notably higher than that of representative multimodal models. PGMCR attains a CoNLL F1 score of 62.53% on the CR task, thereby surpassing existing state-of-the-art methods. These results provide validation of the effectiveness of the proposed framework in cross-modal named entity recognition and co-reference resolution, thus providing new insights for multimodal semantic understanding in complex scenarios.

Keywords:

Named entity recognition

Coreference resolution

Generative

Bert

Object detection

Multimodal

Visual grounding

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.