.png){kind=icon}
.png){kind=icon}
Return
Speculative Decoding on the SN40L Reconfigurable Dataflow Unit
.png){kind=icon}
PRE
AI
DOI：10.1109/MM.2025.3592570.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Speculative decoding has emerged as a promising optimization strategy to accelerate generative artificial intelligence (AI) inference. This technique enhances the autoregressive decoding phase by using a smaller draft model to generate a few tokens, which are then validated by a larger model. However, synchronization overheads on both GPUs and hosts limit the performance gains. This article explores the implementation and optimization potential of batched speculative decoding within SambaNova's SN40L reconfigurable dataflow unit. We achieve more than 75% of the theoretical maximum performance for speculative decoding and delivering a 6x speedup compared to the baseline model. Furthermore, we demonstrate that a single SN40L rack, containing 16 sockets and offering high bandwidth memory bandwidth comparable to the DGX H100, outperforms the latter by up to 1.7x. The techniques and models discussed are deployed in SambaNova's production AI inference cloud, cloud.sambanova.ai, showcasing their tangible impact on large-scale AI applications.
Keywords:
Decoding
Computational modeling
Artificial intelligence
Training
Synchronization
Computer architecture
Throughput
Sockets
Runtime
Production