发现扩散语言模型早期决策关键,提出轻量引导方法提升生成质量。
Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Models

- 通过分析推理过程,揭示非自回归生成中的邻近偏差问题。
- 改进后在多个推理任务上显著优于现有启发式方法。
- 仅用轻量规划器和温度退火,计算开销几乎不变。
基于扩散的语言模型(dLLMs)为非自回归生成提供了并行化与双向上下文建模的潜力。本文系统分析了dLLMs在非自回归解码中的推理动态,发现基于置信度的生成存在严重邻近偏差——去噪顺序过度集中于空间相邻的词元。这种局部依赖导致错误传播,使整个生成轨迹高度依赖初始未掩码位置。基于此,我们提出一种最小干预策略,通过轻量级规划器引导早期词元选择,并结合序列末尾温度退火。在多种推理与规划任务上评估表明,该方法在不增加显著计算开销的前提下,显著优于现有启发式基线。
原文摘要 · Abstract (English)
Diffusion-based language models (dLLMs) have emerged as a promising alternative to autoregressive language models, offering the potential for parallel token generation and bidirectional context modeling. However, harnessing this flexibility for fully non-autoregressive decoding remains an open question, particularly for reasoning and planning tasks. In this work, we investigate non-autoregressive decoding in dLLMs by systematically analyzing its inference dynamics along the temporal axis. Specifically, we uncover an inherent failure mode in confidence-based non-autoregressive generation stemming from a strong proximity bias-the tendency for the denoising order to concentrate on spatially adjacent tokens. This local dependency leads to spatial error propagation, rendering the entire trajectory critically contingent on the initial unmasking position. Leveraging this insight, we present a minimal-intervention approach that guides early token selection, employing a lightweight planner and end-of-sequence temperature annealing. We thoroughly evaluate our method on various reasoning and planning tasks and observe substantial overall improvement over existing heuristic baselines without significant computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。