Integrating Image-Based and Knowledge-Based Representation Learning

doi:10.1109/TCDS.2019.2906685

Integrating Image-Based and Knowledge-Based Representation Learning

2020-06-01

7

OA

AI

DOI：10.1109/TCDS.2019.2906685

Original

Save

View PDF

Abstract

En 中文

A variety of brain areas is involved in language understanding and generation, accounting for the scope of language that can refer to many real-world matters. In this paper, we investigate how regularities among real-world entities impact emergent language representations. Specifically, we consider knowledge bases, which represent entities and their relations as structured triples, and image representations, which are obtained via deep convolutional networks. We combine these sources of information to learn representations of an image-based knowledge representation learning (IKRL) model. An attention mechanism lets more informative images contribute more to the image-based representations. Evaluation results show that the model outperforms all baselines on the tasks of knowledge graph (KG) completion and triple classification. In analyzing the learned models, we found that the structure-based and image-based representations integrate different aspects of the entities and the attention mechanism provides robustness during learning.

Keywords:

Visualization

Knowledge representation

Brain modeling

Task analysis

Head

Knowledge based systems

Computational modeling

Attention mechanisms and development

embodied cognition

generation of representation during development

Integrating Image-Based and Knowledge-Based Representation Learning

Abstract

AI Summary

Journal

Organization