提出新算法提升扩散语言模型的推理能力,解决强化学习训练中的方差难题。
Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
- 通过分解证据下界估计的方差来源,设计低方差采样策略。
- 在数学、推理和编码任务上优于现有基线,性能更稳定。
- 适合需要高效强化学习微调的扩散语言模型研究者。
扩散语言模型(DLMs)支持并行、无序生成与迭代优化,是自回归大模型的灵活替代。然而,由于似然函数不可计算,将强化学习(RL)微调应用于DLMs仍面临挑战。早期工作如diffu-GRPO采用一步去掩码估算词级似然,虽计算高效但偏差严重。更严谨的方法基于序列级似然,以证据下界(ELBO)为代理目标,但因似然评估成本过高,实际应用受限。本文重新审视ELBO估计,解耦其方差来源,提出沿少数关键维度使用快速确定性积分近似以降低方差。基于此,我们提出群扩散策略优化(GDPO),采用简单有效的半确定蒙特卡洛方案,缓解标准双重蒙特卡洛采样下的方差爆炸问题,在严格评估预算下实现可证明更低方差的估计器。实验表明,GDPO在多数数学、推理与编码基准上持续优于预训练检查点,并超越当前最优基线diffu-GRPO。
原文摘要 · Abstract (English)
Diffusion language models (DLMs) enable parallel, order-agnostic generation with iterative refinement, offering a flexible alternative to autoregressive large language models (LLMs). However, adapting reinforcement learning (RL) fine-tuning to DLMs remains an open challenge because of the intractable likelihood. Pioneering work such as diffu-GRPO estimated token-level likelihoods via one-step unmasking. While computationally efficient, this approach is severely biased. A more principled foundation lies in sequence-level likelihoods, where the evidence lower bound (ELBO) serves as a surrogate. Yet, despite this clean mathematical connection, ELBO-based methods have seen limited adoption due to the prohibitive cost of likelihood evaluation. In this work, we revisit ELBO estimation and disentangle its sources of variance. This decomposition motivates reducing variance through fast, deterministic integral approximations along a few pivotal dimensions. Building on this insight, we introduce Group Diffusion Policy Optimization (GDPO), a new RL algorithm tailored for DLMs. GDPO leverages simple yet effective Semi-deterministic Monte Carlo schemes to mitigate the variance explosion of ELBO estimators under vanilla double Monte Carlo sampling, yielding a provably lower-variance estimator under tight evaluation budgets. Empirically, GDPO achieves consistent gains over pretrained checkpoints and outperforms diffu-GRPO, one of the state-of-the-art baselines, on the majority of math, reasoning, and coding benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。