放弃扩散语言模型的任意生成顺序，反而能提升推理能力。
The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
- 不再追求任意顺序生成，改用标准分组相对策略优化
- 在GSM8K上达到89.1%准确率，超越传统灵活策略
- 适合追求高效推理的模型设计者，尤其关注实用性能
扩散大语言模型（dLLMs）打破了传统语言模型从左到右的生成限制，支持任意顺序生成token。直觉上，这种灵活性应带来更优的推理潜力。然而，本文发现，在数学与编程等通用推理任务中，任意顺序生成反而可能限制dLLMs的推理能力。我们观察到，dLLMs倾向于利用顺序灵活性跳过高不确定性的关键token，导致解空间过早坍缩。这一现象促使我们重新思考dLLMs的强化学习方法——以往为保持灵活性而引入的复杂机制（如处理组合轨迹、不可计算似然）可能得不偿失。我们提出JustGRPO：简单舍弃任意顺序，直接使用标准分组相对策略优化（GRPO）。该方法虽极简，但效果显著（如在GSM8K上达89.1%准确率），同时保留了dLLMs的并行解码优势。
原文摘要 · Abstract (English)
Diffusion Large Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs, enabling token generation in arbitrary orders. Intuitively, this flexibility implies a solution space that strictly supersets the fixed autoregressive trajectory, theoretically unlocking superior reasoning potential. However, in this paper, we find that for general reasoning tasks (e.g., mathematics and coding), arbitrary order generation may in fact limit the reasoning potential of dLLMs. We observe that dLLMs tend to exploit this order flexibility to bypass high-uncertainty tokens that are crucial for exploration, which can lead to a premature collapse of solution coverage. This observation motivates a rethink of RL approaches for dLLMs, where considerable complexities, such as handling combinatorial trajectories and intractable likelihoods, are often devoted to preserving this flexibility. We show that effective reasoning can be elicited by simply forgoing arbitrary order and applying standard Group Relative Policy Optimization (GRPO) instead. Our approach, JustGRPO, is minimalist yet surprisingly effective (e.g., 89.1% accuracy on GSM8K) while fully retaining the parallel decoding ability of dLLMs. Project page: https://nzl-thu.github.io/the-flexibility-trap
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。