DySpec: Faster speculative decoding with dynamic token tree structure

doi:10.1007/s11280-025-01344-0

2025-05-08

PRE

DOI：10.1007/s11280-025-01344-0

原文链接

原文求助

摘要

En 中文

尽管推测解码（speculative decoding）作为加速大型语言模型（LLMs）推理的可行方向已崭露头角，其加速效果和可扩展性却严重受限于token接受率。现有方法通常将预测的token组织为独立的链或固定的token树结构，但难以泛化至多样化的查询分布。本文提出DySpec，一种更快的推测解码算法，并引入新颖的动态token树结构。我们首先通过直观和经验性的线索关联了草稿分布（draft distribution）与接受率，并成功证明两者存在强相关性。基于此，我们采用贪心策略在运行时动态扩展token树。理论上，我们证明该方法在温和假设下可实现最优结果。实验表明，DySpec相较于固定树结构能获得更高的接受率与加速比。DySpec能在不同数据分布和模型规模下显著提升吞吐量并降低token生成的延迟，其性能显著优于强基线，包括Specinfer和Sequoia。在低温度设置下，DySpec可在Llama2-70B模型上将吞吐量提升至9.1倍，延迟降低至9.4倍；而在高温度设置下，DySpec仍能将吞吐量提升至6.21倍，尽管此时草稿模型每步推测多个token的难度有所增加。

Keyword:

Artificial intelligence

Large language models

Inference acceleration

Speculative decoding

期刊

World Wide Web-Internet and Web Information Systems

IF:

3.4

论文数:

被引数:

2.3K

机构

peking university

学者数:

11.9W

论文数: 8.7W

被引数: 146

引用论文

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning

IF0

2016-01-01

OAAI

Ramesh Nallapati; Bowen Zhou; Cicero dos Santos; Caglar Gulcehre; Bing Xiang

A data structure for dynamic trees

Proceedings of the thirteenth annual ACM symposium on Theory of computing - STOC '81

IF0

1981-01-01

OAAI

Daniel D. Sleator; Robert Endre Tarjan

DeepSpeed

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

IF0

2020-08-20

PREAI

Jeff Rasley; Samyam Rajbhandari; Olatunji Ruwase; Yuxiong He

EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees EAGLE-2：使用动态草稿树加速语言模型推理

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

IF0

2024-01-01

PREAI

Yuhui Li; Fangyun Wei; Chao Zhang; Hongyang Zhang

REST: Retrieval-Based Speculative Decoding REST：基于检索的投机性解码

Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

IF0

2024-01-01

OAAI

Zhenyu He; Zexuan Zhong; Tianle Cai; Jason Lee; Di He

SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3

IF0

2024-04-27

OAAI

Xupeng Miao; Gabriele Oliaro; Zhihao Zhang; Xinhao Cheng; Zeyu Wang; Zhengxin Zhang; Rae Ying Yee Wong; Alan Zhu; Lijie Yang; Xiaoxiang Shi; Chunan Shi; Zhuoming Chen; Daiyaan Arfeen; Reyna Abhyankar; Zhihao Jia

K-BERT: Enabling Language Representation with Knowledge Graph

Proceedings of the AAAI Conference on Artificial Intelligence

IF0

2020-04-03

OAAI

Weijie Liu; Peng Zhou; Zhe Zhao; Zhiruo Wang; Qi Ju; Haotang Deng; Ping Wang

Triton: an intermediate language and compiler for tiled neural network computations

Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages

IF0

2019-06-22

PREAI

Philippe Tillet; H. T. Kung; David Cox

没有更多内容