arXiv:2606.07710cs.LGcs.AI2026-06

通过动态切换生成方式,让大模型推理速度提升近70%。

WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

  • 根据每个词的复杂度,自动选择自回归或扩散方式生成。
  • 相比最优现有方法,吞吐量最高提升69.6%。
  • 适合需要高速推理的智能体任务,如复杂问答与规划。

大语言模型的自回归特性是推理中的主要瓶颈,尤其在复杂的智能体工作负载中。尽管推测解码(SD)能加速推理,但现有方法依赖静态的草稿范式——要么用自回归模型进行推理,要么用基于扩散的并行模型生成结构化输出。我们发现,单个序列内草稿准确率波动极大,静态范式和粗粒度路由导致性能未被充分释放。为此,我们提出WhiFlash,首个实现自回归与扩散并行草稿统一的跨范式推测解码方法,采用细粒度的词级别控制器,结合轻量级熵策略或可学习神经策略,灵活平衡预期词数收益与延迟。为支持高频切换,引入新的缓存优化技术:懒惰追赶(Lazy Catch-up)与仅键值预填充(KV-only Prefill),将切换开销控制在每轮延迟的7%以下。通过融合两种根本不同的草稿架构优势,WhiFlash显著提升接受长度,在类别特定任务上相较当前最优的自回归EAGLE-3提升69.6%,较扩散基的DFlash提升37.3%。

原文摘要 · Abstract (English)

The autoregressive nature of large language models (LLMs) remains a significant bottleneck for inference, particularly in complex agentic workloads. While speculative decoding (SD) accelerates inference, current approaches rely on static drafting paradigms, utilising either autoregressive drafting models for reasoning or diffusion-based parallel drafting models for structured outputs. We empirically find that drafting accuracy fluctuates dramatically within a single sequence, leaving significant performance unrealised by static paradigms and coarse-grained routing. To address this volatility, we introduce WhiFlash, the first cross-paradigm SD method that unifies autoregressive and diffusion-based parallel drafting under a single token-level controller. WhiFlash adopts a fine-grained routing mechanism that employs either a lightweight entropy-based or a learned neural policy, both parametrised to provide a tunable balance between expected token gain and latency. To make high-frequency switching computationally viable, we introduce novel cache-management optimisations, Lazy Catch-up and KV-only Prefill, reducing switching overhead to below 7% of per-round latency. By capitalising on the complementary strengths of fundamentally distinct drafting architectures, WhiFlash achieves significantly higher acceptance lengths, yielding category-specific throughput gains of up to 69.6% over the state-of-the-art autoregressive EAGLE-3 and 37.3% over the diffusion-based DFlash.

推理加速推测解码大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。