arXiv:2510.04767cs.LG2025-10中稿 · ICLR被引 46

提出首个针对扩散型大模型并行解码的基准测试,揭示速度与质量的深层权衡。

ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs

  • 构建 ParallelBench 基准,聚焦真实任务中并行解码的缺陷
  • 实测发现并行解码在复杂任务下生成质量显著下降
  • 适合关注推理效率与生成质量平衡的研究者

尽管大多数自回归大模型受限于逐个解码,扩散型大模型(dLLMs)因其并行解码潜力而备受关注,有望大幅加速推理。然而,dLLMs 中的条件独立性假设导致并行解码忽略词元依赖关系,当依赖性强时必然降低生成质量。现有研究大多忽视此问题,且标准基准(如数学和编程任务)不足以捕捉并行解码带来的质量退化。为此,我们首先进行信息论分析,再通过可解析的合成列表操作案例研究,从数据分布与解码策略角度提供量化洞见,揭示并行解码的根本局限。基于此,我们提出 ParallelBench——首个专为 dLLMs 设计的基准,包含对人类和自回归模型易、但对 dLLMs 并行解码极难的真实任务。利用该基准,我们系统评估 dLLMs 与自回归模型,发现:(i) dLLMs 在并行解码下在现实场景中质量严重下降;(ii) 当前并行解码策略无法根据任务难度动态调整并行程度,难以在不牺牲质量的前提下实现有效提速。研究凸显了亟需创新解码方法以突破当前速度-质量权衡。我们开源该基准,助力高效 dLLMs 的发展。

原文摘要 · Abstract (English)

While most autoregressive LLMs are constrained to one-by-one decoding, diffusion LLMs (dLLMs) have attracted growing interest for their potential to dramatically accelerate inference through parallel decoding. Despite this promise, the conditional independence assumption in dLLMs causes parallel decoding to ignore token dependencies, inevitably degrading generation quality when these dependencies are strong. However, existing works largely overlook these inherent challenges, and evaluations on standard benchmarks (e.g., math and coding) are not sufficient to capture the quality degradation caused by parallel decoding. To address this gap, we first provide an information-theoretic analysis of parallel decoding. We then conduct case studies on analytically tractable synthetic list operations from both data distribution and decoding strategy perspectives, offering quantitative insights that highlight the fundamental limitations of parallel decoding. Building on these insights, we propose ParallelBench, the first benchmark specifically designed for dLLMs, featuring realistic tasks that are trivial for humans and autoregressive LLMs yet exceptionally challenging for dLLMs under parallel decoding. Using ParallelBench, we systematically analyze both dLLMs and autoregressive LLMs, revealing that: (i) dLLMs under parallel decoding can suffer dramatic quality degradation in real-world scenarios, and (ii) current parallel decoding strategies struggle to adapt their degree of parallelism based on task difficulty, thus failing to achieve meaningful speedup without compromising quality. Our findings underscore the pressing need for innovative decoding methods that can overcome the current speed-quality trade-off. We release our benchmark to help accelerate the development of truly efficient dLLMs.

扩散模型并行解码推理优化基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。