Beyond Word Embeddings: Heterogeneous Prior Knowledge Driven Multi-Label Image Classification

doi:10.1109/TMM.2022.3171095

2023-01-01

PRE

DOI：10.1109/TMM.2022.3171095

原文链接

原文求助

摘要

En 中文

Multi-Label Image Classification (MLIC) is a fundamental yet challenging task which aims to recognize multiple labels from given images. The key to solve MLIC lies in how to accurately model the correlation between labels. Recent studies often adopt Graph Convolutional Network (GCN) to model label dependencies with word embeddings as prior knowledge. However, classical word embeddings typically contain redundant information due to the imperfect distributional hypothesis it relies on, which may degrade model generalizability. To tackle this problem, we propose a novel deep learning framework termed Visual-Semantic based Graph Convolutional Network (VSGCN), which alleviates the negative impact of redundant information by utilizing heterogeneous sources of prior knowledge. Specifically, we construct both visual prototype and semantic prototype for each label as heterogeneous prior label representations, which are further mapped to multi-label classifiers via two Multi-Head GCNs separately. The Multi-Head GCN mechanism proposed in this paper aims to guide the information propagation between prototypes for each label, which constructs multiple correlation graphs to simultaneously model the label correlation in different subspaces. Notably, we alleviate the negative influence of needless information by decreasing the inconsistency of predictions that come from visual space and semantic space. Extensive experiments conducted on various multi-label image datasets demonstrate the superiority of our proposed method.

Keyword:

Correlation

Semantics

Prototypes

Visualization

Transformers

Knowledge engineering

Image classification

Multi-label image classification

heterogeneous prior knowledge

graph convolutional network

multi-head attention