Accelerating Sparse Transformer Inference on GPU

Wenhao Dai

doi:10.1145/3774934.3786434

Accelerating Sparse Transformer Inference on GPU

2026-01-01

0

PRE

AI

DOI：10.1145/3774934.3786434

Original

Original request for help

Save

Abstract

En 中文

Large language models (LLMs) are popular around the world due to their powerful understanding capabilities. As the core component of LLMs, accelerating Transformer through parallelization has gradually become a hot research topic. Mask layers introduce sparsity into Transformer to reduce calculations. However, previous works rarely focus on the performance optimization of sparse Transformer. In addition, current static operator fusion schemes fail to adapt to diverse application scenarios. To address the above problems, we propose STOF, a framework that incorporates optimizations for Sparse Transformer that enables flexible masking and Operator Fusion on GPU. For multi-head attention (MHA) structure, STOF maps the computation to row-wise or blockwise kernels with unique storage formats according to analytical modeling. For downstream operators, STOF maps the fusion scheme to compilation templates and determines the optimal running configuration through two-stage searching.The experimental results show that compared to the stateof-the-art work, STOF achieves maximum speedups of 1.6x in MHA computation and 1.4x in end-to-end inference.

Keywords:

GPU

Sparse Transformer

Multi-head Attention

Operator Fusion

Accelerating Sparse Transformer Inference on GPU

Abstract

Journal

Organization