Human-Centric Image Captioning

doi:10.1016/j.patcog.2022.108545

2022-06-01

PRE

DOI：10.1016/j.patcog.2022.108545

原文链接

原文求助

摘要

En 中文

In this paper, we propose a new topic, Human-Centric Captioning, to mainly describe the human behavior in an image. Human activities and relationships are the primary objectives of visual understanding in daily applications. However, existing image captioning systems cannot differently treat humans and other objects, which limits the ability to understand and describe diverse human activities. As the first explorer of this new task, we build a novel Human-Centric COCO dataset concentrating on humans. Accordingly, we propose a novel Human-Centric Captioning Model (HCCM) that focuses on human-centric feature hierarchization and sentence generation. Specifically, our model first utilizes human body part level knowledge to hierarchize the image features and then applies a novel three-branch captioning model to process these hierarchical features independently to calibrate the descriptions of human actions. Comprehensive experiments demonstrate that our HCCM achieves the state-of-the-art performance with BLEU-4, CIDEr and SPICE scores of 41.5, 127.3, 23.5 respectively. Dataset and code are publicly available at https://github.com/JohnDreamer/HCCM/. (c) 2022 Elsevier Ltd. All rights reserved.

Keyword:

Human-centric

Image captioning

Feature hierarchization

期刊

Pattern Recognition

IF:

7.6

论文数:

1.3W

被引数:

4.5W

机构

shanghai jiao tong university

学者数:

15.7W

论文数: 11.7W

被引数: 159

引用论文

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations 视觉基因组: 使用众包密集图像注释连接语言和视觉

INTERNATIONAL JOURNAL OF COMPUTER VISION

IF9.3

2017-02-06

3.1K

OAAI

Krishna, Ranjay; Zhu, Yuke; Groth, Oliver; Johnson, Justin; Hata, Kenji; Kravitz, Joshua; Chen, Stephanie; Kalantidis, Yannis; Li, Li-Jia; Shamma, David A.; Bernstein, Michael S.; Li Fei-Fei

Field-weakening performance of brushless synchronous AC motor drives 无刷同步交流电机驱动器的弱磁性能

IEE Proceedings - Electric Power Applications

IF0

1994-01-01

PREAI

W.L. Soong

Enhancing the alignment between target words and corresponding frames for video captioning

PATTERN RECOGNITION

IF7.6

2021-03-01

PREAI

Tu, Yunbin; Zhou, Chang; Guo, Junjun; Gao, Shengxiang; Yu, Zhengtao