通过分层验证提升大模型推理速度，减少无效计算。
Hierarchical Verification of Speculative Beams for Accelerating LLM Inference
- 构建分层验证树，优先验证高概率候选序列。
- 相比现有方法，推理时间与能耗显著降低。
- 无需重训练即可接入主流模型，适合部署优化。
大型语言模型在自然语言处理任务中表现卓越，但因其自回归特性，推理效率始终面临挑战。尽管推测解码和束搜索能带来明显改进，传统方法仍按顺序验证草稿序列，缺乏优先级，造成不必要的计算开销。本文提出分层验证树（HVT）框架，通过重构推测束搜索机制，优先验证高似然草稿并实现对次优候选的早期剪枝。建立了理论基础和形式化验证-剪枝算法，确保正确性与高效性。该框架可无缝集成至标准大模型推理流程，无需重新训练或架构修改。跨多个数据集和模型的实验表明，HVT持续优于现有推测解码方案，在保持甚至提升输出质量的同时，大幅降低推理时间和能耗。研究结果表明，分层验证策略是加速大模型推理的新方向。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable success across diverse natural language processing tasks but face persistent challenges in inference efficiency due to their autoregressive nature. While speculative decoding and beam sampling offer notable improvements, traditional methods verify draft sequences sequentially without prioritization, leading to unnecessary computational overhead. This work proposes the Hierarchical Verification Tree (HVT), a novel framework that restructures speculative beam decoding by prioritizing high-likelihood drafts and enabling early pruning of suboptimal candidates. Theoretical foundations and a formal verification-pruning algorithm are developed to ensure correctness and efficiency. Integration with standard LLM inference pipelines is achieved without requiring retraining or architecture modification. Experimental evaluations across multiple datasets and models demonstrate that HVT consistently outperforms existing speculative decoding schemes, achieving substantial reductions in inference time and energy consumption while maintaining or enhancing output quality. The findings highlight the potential of hierarchical verification strategies as a new direction for accelerating large language model inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。