提出LEAP方法,让扩散语言模型提前识别可并行解码的词元,显著提速。
LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection

- 通过未来上下文过滤和多序列叠加,检测早期收敛词元
- 平均减少30%去噪步数,在GSM8K上达7.2词元/步
- 无需训练、可直接插件使用,适合高效推理场景
扩散语言模型(dLLMs)因其高度并行处理潜力受到广泛关注。现有dLLMs的并行能力依赖于高置信度下的条件独立性假设,但为保证精度所需严苛的置信度阈值严重限制了并行扩展性。通过系统性的词元级统计分析,我们发现大量词元在去噪过程中早期即正确收敛,却未达到标准置信度阈值,表明当前基于置信度的判据过于保守。为此,我们提出LEAP(Lookahead Early-Convergence Token Detection for Accelerated Parallel Decoding)。LEAP是一种无需训练、可即插即用的方法,利用未来上下文过滤与多序列叠加技术检测早期收敛词元。通过验证早期收敛与正确性的对齐关系,实现这些词元的可靠提前解码。跨多个领域的基准测试显示,LEAP显著降低推理延迟与解码步数。相比基于置信度的解码,平均去噪步数减少约30%。在GSM8K数据集上,结合dParallel时,解码速度达7.2词元/步,同时保持模型精度。LEAP有效打破对高置信度先验的依赖,提供一种全新的并行解码范式。
原文摘要 · Abstract (English)
Diffusion Language Models (dLLMs) have garnered significant attention for their potential in highly parallel processing. The parallel capabilities of existing dLLMs stem from the assumption of conditional independence at high confidence levels, which ensures negligible discrepancy between the marginal and joint distributions. However, the stringent confidence thresholds required to preserve accuracy severely constrain the scalability of parallelism. Through systematic token-level statistical analysis, we reveal that a substantial proportion of tokens converge to their correct predictions early in the denoising process yet fail to reach standard confidence thresholds, confirming that current confidence-based criteria are overly conservative. In response, we introduce LEAP (Lookahead Early-Convergence Token Detection for Accelerated Parallel Decoding). LEAP is a training-free, plug-and-play method that leverages future context filtering and multi-sequence superposition to detect early-converging tokens. By validating the alignment between early convergence and correctness, we enable reliable early decoding of these tokens. Benchmarking across diverse domains demonstrates that LEAP significantly lowers inference latency and decoding steps. Compared to confidence-based decoding, the average number of denoising steps is reduced by about 30%. On the GSM8K dataset, combining LEAP with dParallel accelerates decoding to 7.2 tokens per step while preserving model precision. LEAP effectively breaks the reliance on high-confidence priors, offering a novel paradigm for parallel decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。