动态选择解码方式，让大模型推理更快更准
Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement
- 根据每步需求自动选串行或并行解码
- 相比自回归推理最高提速2.12倍，优于Medusa框架
- 适合追求高效且高质量生成的开发者使用
大语言模型在推理时常因自回归解码导致速度瓶颈。尽管并行解码可提升效率，但现有框架存在两大问题：（1）解码头难以平衡预测精度与执行并行性；（2）并行解码并非普适方案，某些复杂步骤反而引入额外开销。为此，我们提出Cerberus，一种自适应并行解码框架，引入门控机制使模型在每一步动态选择最优解码策略，并设计新型解码头，在保持执行并行的同时引入序列知识。实验表明，Cerberus相比自回归解码最高实现2.12倍加速，相较于领先并行解码框架Medusa，加速提升10%–30%，且生成质量更优。
原文摘要 · Abstract (English)
Large language models (LLMs) often face a bottleneck in inference speed due to their reliance on auto-regressive decoding. Recently, parallel decoding has shown significant promise in enhancing inference efficiency. However, we have identified two key issues with existing parallel decoding frameworks: (1) decoding heads fail to balance prediction accuracy and the parallelism of execution, and (2) parallel decoding is not a universal solution, as it can bring unnecessary overheads at some challenging decoding steps. To address these issues, we propose Cerberus, an adaptive parallel decoding framework introduces the gating mechanism to enable the LLMs to adaptively choose appropriate decoding approaches at each decoding step, along with introducing a new paradigm of decoding heads that introduce the sequential knowledge while maintaining execution parallelism. The experiment results demonstrate that the Cerberus can achieve up to 2.12x speed up compared to auto-regressive decoding, and outperforms one of the leading parallel decoding frameworks, Medusa, with a 10% - 30% increase in acceleration and superior generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。