Graph Receptive Transformer Encoder for Text Classification

Arda Can Aras

doi:10.1109/TSIPN.2024.3380362

2024-01-01

PRE

Arda Can Aras

Tuna Alikaşifoğlu

Aykut Koç *

DOI：10.1109/TSIPN.2024.3380362

原文链接

原文求助

摘要

En 中文

By employing attention mechanisms, transformers have made great improvements in nearly all NLP tasks, including text classification. However, the context of the transformer's attention mechanism is limited to single sequences, and their fine-tuning stage can utilize only inductive learning. Focusing on broader contexts by representing texts as graphs, previous works have generalized transformer models to graph domains to employ attention mechanisms beyond single sequences. However, these approaches either require exhaustive pre-training stages, learn only transductively, or can learn inductively without utilizing pre-trained models. To address these problems simultaneously, we propose the Graph Receptive Transformer Encoder (GRTE), which combines graph neural networks (GNNs) with large-scale pre-trained models for text classification in both inductive and transductive fashions. By constructing heterogeneous and homogeneous graphs over given corpora and not requiring a pre-training stage, GRTE can utilize information from both large-scale pre-trained models and graph-structured relations. Our proposed method retrieves global and contextual information in documents and generates word embeddings as a by-product of inductive inference. We compared the proposed GRTE with a wide range of baseline models through comprehensive experiments. Compared to the state-of-the-art, we demonstrated that GRTE improves model performances and offers computational savings up to similar to 100x.

Keyword:

BERT

graph convolutional networks (GCNs)

graph neural networks (GNNs)

inductive

text classification

transductive

transformers