arXiv:2510.13161cs.CL2025-10被引 1

通过双路并行推测加速大模型推理,突破速度与准确率的权衡瓶颈。

Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference

  • 利用双执行管道并行推测目标模型与草稿模型的后续路径。
  • 在14B到66B参数模型上实现2.8至5.8倍的端到端加速。
  • 适合追求高吞吐、低延迟的大模型部署场景。

推测解码通过草稿模型前瞻生成内容以加速大语言模型推理,但其性能受限于自回归草稿生成的开销:增大草稿规模虽提高接受率,却引入额外延迟,加剧速度与准确率的权衡。现有方法(Medusa、Hydra、EAGLE)部分降低草稿成本,但或降低接受率,或引入开销限制扩展性。本文提出镜像推测解码(Mirror-SD),打破延迟-接受率的权衡。Mirror-SD 从早期退出信号并行启动完整分支推演,与目标模型后缀同步执行,并显式映射计算至异构加速器(GPU与NPU),利用跨设备并行。草稿为目标模型推测后续内容以验证,同时目标模型为草稿推测修正路径,将推测转化为两条互补执行流。为进一步降低草稿延迟而不削弱接受语义,引入推测流式输出,使草稿每步可生成多个词元。这一并行异构执行与多词元推测流式的双重策略,推动推测解码逼近理想状态——高接受率、低开销。在包含14B至66B参数的服务器级模型上的SpecBench测试中,Mirror-SD实现稳定端到端加速,各任务下获得2.8x–5.8x的墙时加速,平均相对性能优于最强基线EAGLE3达30%。

原文摘要 · Abstract (English)

Speculative decoding accelerates LLM inference by using a draft model to look ahead, but gains are capped by the cost of autoregressive draft generation: increasing draft size elevates acceptance rates but introduces additional latency overhead exacerbating the speed-accuracy tradeoff. Prior methods (Medusa, Hydra, EAGLE) partially reduce draft cost but either degrade acceptance or introduce overheads that limit scaling. We present Mirror Speculative Decoding (Mirror-SD), an inference algorithm that breaks the latency-acceptance tradeoff. Mirror-SD launches branch-complete rollouts from early-exit signals in parallel with the target model's suffix and explicitly maps computation across heterogeneous accelerators (GPU and NPU) to exploit cross-device parallelism. The draft speculates forward continuations for the target to verify, while the target simultaneously speculates correction paths for the draft, converting speculation into two complementary execution pipelines. To further cut draft latency without weakening acceptance semantics, we add speculative streaming so the draft emits multiple tokens per step. This dual strategy of parallel heterogeneous execution plus multi-token speculative streaming pushes speculative decoding toward its ideal regime of high acceptance with low overhead. On SpecBench with server-scale models from 14B to 66B parameters, Mirror-SD delivers consistent end-to-end gains, achieving 2.8x-5.8x wall-time speedups across diverse tasks and a 30% average relative improvement over the strongest baseline, EAGLE3.

大模型推理推测解码并行计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。