针对大模型推理中候选生成验证效率不均的问题，提出自适应分配验证资源的新方法。
HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding
- 根据候选预测的不确定性动态分配验证资源，避免无效计算。
- 在五项基准上平均提速4.24倍，且输出分布完全一致。
- 无需重训练，可与现有优化技术兼容，适合实际部署。
自回归解码因序列依赖性限制了大语言模型的推理吞吐量。推测解码通过并行验证多个预测标记缓解此问题，但其效率仍受我们定义的验证异质性制约——不同推测候选的验证难度不一。实践中，少数高置信度预测贡献了大部分成功验证，而现有方法对所有候选同等处理，造成冗余计算。我们提出HeteroSpec，一种适应异质性的推测解码框架，按候选不确定性比例分配验证资源。HeteroSpec使用轻量级熵量化器估计验证复杂度，通过数据驱动分层策略划分候选，并协同优化推测深度与剪枝阈值。在五个基准和四个LLM上，HeteroSpec相比EAGLE-3等先进方法平均实现4.24倍加速，且保持精确输出分布。关键的是，HeteroSpec无需模型重训练，兼容其他推理优化，是提升推测解码效率的实用方向。
原文摘要 · Abstract (English)
Autoregressive decoding inherently limits the inference throughput of Large Language Model (LLM) due to its sequential dependency. Speculative decoding mitigates this by verifying multiple predicted tokens in parallel, but its efficiency remains constrained by what we identify as verification heterogeneity -- the uneven difficulty of verifying different speculative candidates. In practice, a small subset of high-confidence predictions accounts for most successful verifications, yet existing methods treat all candidates uniformly, leading to redundant computation. We present HeteroSpec, a heterogeneity-adaptive speculative decoding framework that allocates verification effort in proportion to candidate uncertainty. HeteroSpec estimates verification complexity using a lightweight entropy-based quantifier, partitions candidates via a data-driven stratification policy, and dynamically tunes speculative depth and pruning thresholds through coordinated optimization. Across five benchmarks and four LLMs, HeteroSpec delivers an average 4.24$\times$ decoding speedup over state-of-the-art methods such as EAGLE-3, while preserving exact output distributions. Crucially, HeteroSpec requires no model retraining and remains compatible with other inference optimizations, making it a practical direction for improving speculative decoding efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。