.png){kind=icon}
Return
Accelerating Sparse Transformer Inference on GPU
.png){kind=icon}
PRE
AI
DOI：10.1145/3774934.3786434.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Large language models (LLMs) are popular around the world due to their powerful understanding capabilities. As the core component of LLMs, accelerating Transformer through parallelization has gradually become a hot research topic. Mask layers introduce sparsity into Transformer to reduce calculations. However, previous works rarely focus on the performance optimization of sparse Transformer. In addition, current static operator fusion schemes fail to adapt to diverse application scenarios. To address the above problems, we propose STOF, a framework that incorporates optimizations for Sparse Transformer that enables flexible masking and Operator Fusion on GPU. For multi-head attention (MHA) structure, STOF maps the computation to row-wise or blockwise kernels with unique storage formats according to analytical modeling. For downstream operators, STOF maps the fusion scheme to compilation templates and determines the optimal running configuration through two-stage searching.The experimental results show that compared to the stateof-the-art work, STOF achieves maximum speedups of 1.6x in MHA computation and 1.4x in end-to-end inference.
Keywords:
GPU
Sparse Transformer
Multi-head Attention
Operator Fusion
Journal
Organization
View more organizations
Cited Papers
.png){kind=icon}
ShareSaveShareSaveRaptor-T: A Fused and Memory-Efficient Sparse Transformer for Long and Variable-Length Sequences
PREAI
ShareSaveShareSaveShareSave
View more