用轻量分类器动态构建候选词树，降低大模型推理延迟
C2T: A Classifier-Based Tree Construction Method in Speculative Decoding
- 用分类器结合额外特征预测候选词置信度，动态生成词树
- 相比SOTA方法减少25%候选词数，且接受长度不降反升
- 适合追求高效推理的部署场景，尤其大模型应用
大型语言模型（LLMs）规模的增长加剧了推理延迟和计算开销。推测解码方法旨在缓解这些问题，但在候选词树构建和候选词验证方面常存在效率瓶颈。现有方法如链式模式、静态树和动态树，在准确生成用于验证的候选词树方面存在局限。本文提出一种新方法C2T，采用轻量级分类器动态生成并剪枝词树。该分类器不仅考虑常用联合概率，还引入额外特征变量，对每个草稿词预测置信度，以判断其是否为验证候选。在多个基准测试中，该方法优于EAGLE-2等当前最优方法，总候选词数减少25%，同时保持或提升接受长度。
原文摘要 · Abstract (English)
The growing scale of Large Language Models (LLMs) has exacerbated inference latency and computational costs. Speculative decoding methods, which aim to mitigate these issues, often face inefficiencies in the construction of token trees and the verification of candidate tokens. Existing strategies, including chain mode, static tree, and dynamic tree approaches, have limitations in accurately preparing candidate token trees for verification. We propose a novel method named C2T that adopts a lightweight classifier to generate and prune token trees dynamically. Our classifier considers additional feature variables beyond the commonly used joint probability to predict the confidence score for each draft token to determine whether it is the candidate token for verification. This method outperforms state-of-the-art (SOTA) methods such as EAGLE-2 on multiple benchmarks, by reducing the total number of candidate tokens by 25% while maintaining or even improving the acceptance length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。