为扩散语言模型设计序列级强化学习方法，提升生成质量。
Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
- 将整段生成视为单一动作，用变分下界近似序列概率
- 在计数任务上比基线提升20-40分，数学与编程任务也持续优化
- 适合研究扩散模型与强化学习结合的学者
强化学习在自回归语言模型中表现优异，但将其应用于扩散大语言模型（dLLMs）面临根本挑战：难以进行似然估计。自回归模型天然提供用于逐标记强化学习目标（如GRPO）的条件概率，而dLLMs通过迭代非自回归去噪生成序列，缺乏这种因子分解结构。为此，我们提出基于证据下界（ELBO）的序列级策略优化（ESPO），将整个序列生成视为单一动作，并使用ELBO作为可计算的序列级似然代理。该方法引入逐标记重要性比率归一化和鲁棒的KL散度估计，确保大规模训练的稳定性。在数学推理、编程和规划任务上的大量实验表明，ESPO显著优于逐标记基线，在计数任务上提升20-40分，同时在数学与编程基准上保持一致增益。本方法确立了序列级优化在dLLM强化学习中的原则性与有效性。代码已公开于https://github.com/ML-GSAI/ESPO。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has proven highly effective for autoregressive language models, but adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges. The core difficulty lies in likelihood approximation: while autoregressive models naturally provide token-level conditional probabilities essential for token-level RL objectives (e.g., GRPO), dLLMs generate sequences through iterative non-autoregressive denoising steps that lack this factorization. To address this fundamental mismatch, we propose ELBO-based Sequence-level Policy Optimization (ESPO), a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy. Our method incorporates per-token normalization of importance ratios and robust KL-divergence estimation to ensure stable large-scale training. Extensive experiments on mathematical reasoning, coding, and planning tasks demonstrate that ESPO significantly outperforms token-level baselines, achieving dramatic improvements of 20-40 points on the Countdown task, while maintaining consistent gains on math and coding benchmarks. Our approach establishes sequence-level optimization as a principled and empirically effective paradigm for RL in dLLMs. Our code is available at https://github.com/ML-GSAI/ESPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。