.png){kind=icon}
.png){kind=icon}
Return
PredLM: A Sparse Large Language Model Decoding Accelerator With Hardware-Efficient Zero-Activation Prediction
.png){kind=icon}
PRE
AI
DOI：10.1109/TCSII.2026.3690531.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
This brief presents PredLM, a hardware-algorithm co-designed accelerator that aggressively reduces external memory access (EMA) to achieve real-time large language model (LLM) decoding. First, we introduce a simple but effective Key-Value (KV) cache management policy so that the size of KV cache is fixed regardless of the context length, significantly reducing the EMA for the KV cache. Second, we propose a dynamic zero-activation prediction scheme for feed-forward networks, allowing the hardware to dynamically skip unnecessary multiply-accumulate operations and weight fetches. To seamlessly support these predictive algorithms, PredLM features a homogeneous processing element array able to support the execution of both structural sparse and dense vector-matrix multiplications. Implemented in a 28-nm CMOS technology and evaluated with the OPT-1.3B and Llama2-7B models, PredLM achieves a time-per-output-token of 1.76 and 9.41 ms/token and an energy efficiency of 19.77 and 105.09 mJ/token with negligible perplexity degradation, delivering up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$18.1\times $ </tex-math></inline-formula> higher throughput and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.92\times $ </tex-math></inline-formula> better energy efficiency compared to state-of-the-art LLM accelerators.
Keywords:
Large language models (LLMs)
key-value (KV) cache reduction
zero-activation prediction
Journal
I
IF:
0
Papers:
153
Citations:
0