提出SPG方法，让扩散语言模型更好学会人类偏好。
SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
- 用上下界夹住真实对数似然，减少强化学习偏差
- 在GSM8K等任务上比现有方法提升27%准确率
- 适合研究扩散模型与强化学习结合的学者
扩散大语言模型（dLLMs）因其可并行生成多个词元，正成为自回归模型的有效替代。然而，通过强化学习对齐人类偏好或特定任务奖励时，由于其不可计算的对数似然，难以直接应用标准策略梯度方法。先前工作使用证据下界（ELBO）等代理方法，但单侧近似会引入显著策略梯度偏差。为此，我们提出夹心策略梯度（SPG），同时利用真实对数似然的上界和下界。实验表明，SPG显著优于基于ELBO或单步估计的基线方法：在GSM8K上准确率提升3.6%，在MATH500上提升2.6%，在Countdown上提升18.4%，在Sudoku上提升27.0%。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) are emerging as an efficient alternative to autoregressive models due to their ability to decode multiple tokens in parallel. However, aligning dLLMs with human preferences or task-specific rewards via reinforcement learning (RL) is challenging because their intractable log-likelihood precludes the direct application of standard policy gradient methods. While prior work uses surrogates like the evidence lower bound (ELBO), these one-sided approximations can introduce significant policy gradient bias. To address this, we propose the Sandwiched Policy Gradient (SPG) that leverages both an upper and a lower bound of the true log-likelihood. Experiments show that SPG significantly outperforms baselines based on ELBO or one-step estimation. Specifically, SPG improves the accuracy over state-of-the-art RL methods for dLLMs by 3.6% in GSM8K, 2.6% in MATH500, 18.4% in Countdown and 27.0% in Sudoku.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。