解决扩散语言模型生成中质量与探索的权衡难题
Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models
- 用独立M-H采样器平衡生成质量和路径探索
- 在MATH500等基准上提升单样本与多样本性能
- 适合需要高质量推理路径的研究者使用
扩散大语言模型（dLLMs）理论上允许任意顺序解码，可能比自回归模型探索更丰富的推理路径。然而实践中，随机解码常降低生成质量。为缓解此问题，低置信度重掩码通过优先选择高置信度词改善单样本质量（如Pass@$1$），但抑制了探索能力，限制了多样本收益（如Pass@$k$），形成根本性的质量-探索矛盾。本文提供统一解释：低置信度重掩码虽提升局部质量代理指标，却严格约束了序列分布的熵。为此，我们刻画了显式平衡质量与探索的最优分布，并提出一种简单的独立Metropolis-Hastings采样器，在解码过程中近似目标该分布。在MATH500、AIME24/25、HumanEval和MBPP等多个推理基准上的实验表明，该方法优于随机解码与低置信度重掩码，实现更优的质量-探索权衡。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) theoretically permit token decoding in arbitrary order, a flexibility that could enable richer exploration of reasoning paths than autoregressive (AR) LLMs. In practice, however, random-order decoding often hurts generation quality. To mitigate this, low-confidence remasking improves single-sample quality (e.g., Pass@$1$) by prioritizing confident tokens, but it also suppresses exploration and limits multi-sample gains (e.g., Pass@$k$), creating a fundamental quality--exploration dilemma. In this paper, we provide a unified explanation of this dilemma. We show that low-confidence remasking improves a myopic proxy for quality while provably constraining the entropy of the induced sequence distribution. To overcome this limitation, we characterize the optimal distribution that explicitly balances quality and exploration, and develop a simple Independent Metropolis--Hastings sampler that approximately targets this distribution during decoding. Experiments across a range of reasoning benchmarks including MATH500, AIME24/25, HumanEval, and MBPP show that our approach yields better exploration-quality tradeoff than both random and low-confidence remasking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。