CDKM: Common and Distinct Knowledge Mining Network With Content Interaction for Dense Captioning

doi:10.1109/TMM.2024.3407695

CDKM: Common and Distinct Knowledge Mining Network With Content Interaction for Dense Captioning

2024-01-01

3

PRE

AI

DOI：10.1109/TMM.2024.3407695

Original

Original request for help

Save

Abstract

En 中文

The dense captioning task aims at detecting multiple salient regions of an image and describing them separately in natural language. Although significant advancements in the field of dense captioning have been made, there are still some limitations to existing methods in recent years. On the one hand, most dense captioning methods lack strong target detection capabilities and struggle to cover all relevant content when dealing with target-intensive images. On the other hand, current transformer-based methods are powerful but neglect the acquisition and utilization of contextual information, hindering the visual understanding of local areas. To address these issues, we propose a common and distinct knowledge-mining network with content interaction for the task of dense captioning. Our network has a knowledge mining mechanism that improves the detection of salient targets by capturing common and distinct knowledge from multi-scale features. We further propose a content interaction module that combines region features into a unique context based on their correlation. Our experiments on various benchmarks have shown that the proposed method outperforms the current state-of-the-art methods.

Keywords:

Task analysis

Visualization

Feature extraction

Object detection

Knowledge engineering

Correlation

Context modeling

Content interaction

dense captioning

knowledge mining

multimodal

CDKM: Common and Distinct Knowledge Mining Network With Content Interaction for Dense Captioning

Abstract

Journal

Organization