GraKerformer: A Transformer With Graph Kernel for Unsupervised Graph Representation Learning

doi:10.1109/TCYB.2024.3465213

GraKerformer: A Transformer With Graph Kernel for Unsupervised Graph Representation Learning

2024-12-01

0

PRE

AI

DOI：10.1109/TCYB.2024.3465213

Original

Original request for help

Save

Abstract

En 中文

While highly influential in deep learning, especially in natural language processing, the Transformer model has not exhibited competitive performance in unsupervised graph representation learning (UGRL). Conventional approaches, which focus on local substructures on the graph, offer simplicity but often fall short in encapsulating comprehensive structural information of the graph. This deficiency leads to suboptimal generalization performance. To address this, we proposed the GraKerformer model, a variant of the standard Transformer architecture, to mitigate the shortfall in structural information representation and enhance the performance in UGRL. By leveraging the shortest-path graph kernel (SPGK) to weight attention scores and combining graph neural networks, the GraKerformer effectively encodes the nuanced structural information of graphs. We conducted evaluations on the benchmark datasets for graph classification to validate the superior performance of our approach.

Keywords:

Transformers

Kernel

Vectors

Computational modeling

Representation learning

Network architecture

Graph neural networks

Feature extraction

Encoding

Computer architecture

Graph kernel

graph neural networks (GNNs)

structural encoding method

transformer

GraKerformer: A Transformer With Graph Kernel for Unsupervised Graph Representation Learning

Abstract

Journal

Organization