Enhancing text representations separately with entity descriptions

Qinghua Zhao

doi:10.1016/j.neucom.2023.126511

Enhancing text representations separately with entity descriptions

2023-10-01

6

PRE

AI

DOI：10.1016/j.neucom.2023.126511

Original

Original request for help

Save

Abstract

En 中文

Several studies have focused on incorporating language models with entity descriptions to facilitate the model with a better understanding of knowledge. Existing methods usually either integrate descriptions in the pre-training stage by designing description-related tasks, or in the fine-tuning stage by directly appending description strings to the original input, this paper falls into the latter group. We separate entity descriptions from the original text and process them by another lighter module. Specifically, we use the original large model to encode the original input, while the lighter module processes the entity descriptions. We also propose a layer-wise fusion strategy to deeply couple the representations of the input and descriptions. To further improve the fusion of the two representations, we explore two auxil-iary tasks: the entity-description enhancement task and the entity contrastive task. Experiments on (Open Entity, FIGER, FewRel, TACRED, SST) datasets yield respective improvements of (0.9, 1.4, 0.6, 0.5, 0.3). Utilizing ChatGPT as the description embedding method holds the potential for even more promis-ing results. & COPY; 2023 Elsevier B.V. All rights reserved.

Keywords:

Knowledge enhancement

Entity

Entity description

Enhancing text representations separately with entity descriptions

Abstract

Journal

Organization