arXiv:2508.09072cs.CL2025-08

提出无需训练的高效文本生成框架,显著降低大模型推理延迟。

READER: Retrieval-Assisted Drafter for Efficient LLM Inference

  • 通过构建概率高的候选词树实现无损推测解码
  • 单提示下最高提速6.13倍,批量推理达5.92倍
  • 适合追求低延迟的大模型部署与检索增强生成场景

自回归语言模型以因子化似然方式生成文本序列,但其严格顺序解码过程带来了固有的推理延迟下限,成为大规模生成模型可扩展部署的核心障碍。现有加速方法依赖辅助草稿模型或引入额外训练阶段,仅部分缓解了逐标记延迟,却未能解决主导的内存和通信开销。本文提出READER,一种可证明无损的推测解码框架,无需训练辅助草稿模型。READER将推测解码形式化为随机树构建问题,利用自然语言中的经验冗余结构生成高概率候选延续。该方法重新审视草稿树构建问题,在统计性能上显著优于传统随机草稿树方法,并提供复杂度理论分析,刻画了在有限计算与内存资源下的推测解码最优边界。超越以往研究中通常考虑的单序列情形,我们引入内存最优的键值缓存服务策略,保证批处理维度的摊销亚线性开销,使READER可扩展至真实推理负载。全面实验表明,单提示推理最高获得6.13倍的时钟周期加速,批量推理达5.92倍,持续超越现有推测解码基线,同时保持输出完全一致,尤其在检索增强生成流水线中表现更优。结果填补了理论并行极限与实际大模型推理之间的关键差距,暗示了高效部署的新标准。

原文摘要 · Abstract (English)

Autoregressive Language Models instantiate a factorized likelihood over token sequences, yet their strictly sequential decoding process imposes an intrinsic lower bound on inference latency. This bottleneck has emerged as a central obstacle to the scalable deployment of large-scale generative models. Existing acceleration techniques partially mitigate token-level latency by relying on auxiliary draft models or introducing an additional training phase, but fail to address the dominant memory and communication costs. We present READER, a provably lossless speculative decoding framework that bypasses the training of the auxiliary draft model. READER formalizes speculative decoding as a stochastic tree construction problem and exploits the empirical redundancy structure of natural language to generate high-probability candidate continuations. Our method revisits the problem of constructing draft trees, establishing substantial statistical improvements over stochastic draft-tree methods and providing a complexity-theoretic analysis that characterizes the optimality frontier of speculative decoding under bounded computation and memory resources. Beyond the single-sequence regime traditionally considered in prior work, we introduce a memory-optimal key-value cache-serving strategy that guarantees amortized sublinear overhead in the batch dimension, allowing READER to scale to realistic inference workloads. Comprehensive experiments demonstrate up to 6.13x wall-clock speedup on single-prompt inference and up to 5.92x on batched inference, consistently surpassing prior speculative decoding baselines, while preserving exact output equivalence, with even more pronounced gains in retrieval-augmented generation pipelines. Our results close a key gap between theoretical parallelism limits and practical LLM inference, suggesting a new standard for efficient deployment.

大模型推理推测解码效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。