.png){kind=icon}
返回
SADIMM: Accelerating Sparse Attention Using DIMM-Based Near-Memory Processing
.png){kind=icon}
PRE
AI
DOI：10.1109/TC.2024.3500362.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Self-attention mechanism is the performance bottleneck of Transformer based language models. In response, researchers have proposed sparse attention to expedite Transformer execution. However, sparse attention involves massive random access, rendering it as a memory-intensive kernel. Memory-based architectures, such as near-memory processing (NMP), demonstrate notable performance enhancements in memory-intensive applications. Nonetheless, existing NMP-based sparse attention accelerators face suboptimal performance due to hardware and software challenges. On the hardware front, current solutions employ homogeneous logic integration, struggling to support the diverse operations in sparse attention. On the software side, token-based dataflow is commonly adopted, leading to load imbalance after the pruning of weakly connected tokens. To address these challenges, this paper introduces SADIMM, a hardware-software co-designed NMP-based sparse attention accelerator. In hardware, we propose a heterogeneous integration approach to efficiently support various operations within the attention mechanism. This involves employing different logic units for different operations, thereby improving hardware efficiency. In software, we implement a dimension-based dataflow, dividing input sequences by model dimensions. This approach achieves load balancing after the pruning of weakly connected tokens. Compared to NVIDIA RTX A6000 GPU, the experimental results on BERT, BART, and GPT-2 models demonstrate that SADIMM achieves 48x, 35x, 37x speedups and 194x, 202x, 191x energy efficiency improvement, respectively.
Keyword:
Sparse matrices
Hardware
Memory management
Software
Parallel processing
Logic
Bandwidth
Transformers
Faces
DRAM chips
Near-memory processing
sparse attention accelerator
DRAM architecture
software-hardware co-design
期刊
IF:
3.8
论文数:
5.4K
被引数:
9.8K
机构
引用论文
Matrix Effects in the Detection of Pb and Ba in Soils Using Laser-Induced Breakdown Spectroscopy使用激光诱导击穿光谱检测土壤中Pb和Ba的基质效应
PREAI
分享收藏Types of Parent Verbal Responsiveness That Predict Language in Young Children With Autism Spectrum Disorder预测自闭症谱系障碍幼儿语言的父母言语反应类型
OAAI
OAAI分享收藏A Hybrid Prediction Model for Solar Radiation Based on Long Short-Term Memory, Empirical Mode Decomposition, and Solar Profiles for Energy Harvesting Wireless Sensor Networks.png)
EnergiesOAAI
EnergiesIF0
OAAI分享收藏
分享收藏Research on the Efficiency of Wireless Power Transfer System Based on Multi-Auxiliary Transmitting Coils基于多辅助发射线圈的无线电能传输系统效率研究
PREAI
分享收藏
查看更多内容