Vision-Language-Knowledge Co-Embedding for Visual Commonsense Reasoning

JaeYun Lee

doi:10.3390/s21092911

Return

Vision-Language-Knowledge Co-Embedding for Visual Commonsense Reasoning

2021-04-21

JaeYun Lee

Incheol Kim *

DOI：10.3390/s21092911

Original

Save

View PDF

Abstract

En 中文

Visual commonsense reasoning is an intelligent task performed to decide the most appropriate answer to a question while providing the rationale or reason for the answer when an image, a natural language question, and candidate responses are given. For effective visual commonsense reasoning, both the knowledge acquisition problem and the multimodal alignment problem need to be solved. Therefore, we propose a novel Vision-Language-Knowledge Co-embedding (ViLaKC) model that extracts knowledge graphs relevant to the question from an external knowledge base, ConceptNet, and uses them together with the input image to answer the question. The proposed model uses a pretrained vision-language-knowledge embedding module, which co-embeds multimodal data including images, natural language texts, and knowledge graphs into a single feature vector. To reflect the structural information of the knowledge graph, the proposed model uses the graph convolutional neural network layer to embed the knowledge graph first and then uses multi-head self-attention layers to co-embed it with the image and natural language question. The effectiveness and performance of the proposed model are experimentally validated using the VCR v1.0 benchmark dataset.

Keywords:

visual commonsense reasoning

multimodal co-embedding

knowledge graph

graph convolutional network

pretrained multi-head self-attention network

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.