arXiv:2512.22737cs.CL2025-12被引 39

用因果注意力实现高效并行生成,让扩散语言模型跑得比自回归快。

WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference

  • 通过拓扑重排序让掩码位置只依赖已知词,保持因果性同时支持并行。
  • 在推理速度上接近3倍提升,低熵任务下最高达10倍加速。
  • 适合追求高速生成且需兼容现有缓存机制的部署场景。

自回归生成是大语言模型的标准解码方式,但其逐词生成限制了推理时的并行性。扩散语言模型(DLLMs)可通过每步恢复多个掩码词实现并行解码,但在实践中常无法超越优化过的自回归引擎(如vLLM)。主要原因在于多数DLLMs依赖双向注意力,破坏标准前缀键值缓存,导致重复上下文计算,降低效率。本文提出WeDLM,一种完全基于标准因果注意力的扩散解码框架,使并行生成与前缀缓存兼容。核心思想是通过拓扑重排序将已观察词移至物理前缀位置,同时保持其逻辑顺序,使每个掩码位置仅依赖当前已知词且满足因果约束。基于此,我们设计了一种流式解码过程,持续将高置信度词加入递增的左到右前缀,并维持固定并行负载,避免块扩散中常见的停等行为。实验表明,WeDLM在保持强自回归基线质量的同时实现显著加速:在复杂推理基准上接近3倍提速,低熵生成场景下最高达10倍;关键的是,对比对象为使用vLLM服务的自回归基线,在相同部署条件下,证明扩散式解码可实际超越优化的自回归引擎。

原文摘要 · Abstract (English)

Autoregressive (AR) generation is the standard decoding paradigm for Large Language Models (LLMs), but its token-by-token nature limits parallelism at inference time. Diffusion Language Models (DLLMs) offer parallel decoding by recovering multiple masked tokens per step; however, in practice they often fail to translate this parallelism into deployment speed gains over optimized AR engines (e.g., vLLM). A key reason is that many DLLMs rely on bidirectional attention, which breaks standard prefix KV caching and forces repeated contextualization, undermining efficiency. We propose WeDLM, a diffusion decoding framework built entirely on standard causal attention to make parallel generation prefix-cache friendly. The core idea is to let each masked position condition on all currently observed tokens while keeping a strict causal mask, achieved by Topological Reordering that moves observed tokens to the physical prefix while preserving their logical positions. Building on this property, we introduce a streaming decoding procedure that continuously commits confident tokens into a growing left-to-right prefix and maintains a fixed parallel workload, avoiding the stop-and-wait behavior common in block diffusion methods. Experiments show that WeDLM preserves the quality of strong AR backbones while delivering substantial speedups, approaching 3x on challenging reasoning benchmarks and up to 10x in low-entropy generation regimes; critically, our comparisons are against AR baselines served by vLLM under matched deployment settings, demonstrating that diffusion-style decoding can outperform an optimized AR engine in practice.

扩散模型推理加速因果注意力并行生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。