提出新强化学习方法，显著提升扩散型大模型的推理能力。
Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
- 通过分布匹配优化，使扩散模型策略逼近最优奖励导向分布。
- 在多个推理基准上提升超39个百分点，比基线模型高67.97个百分点。
- 适合需要高效推理的场景，尤其适用于无监督微调需求的研究者。
扩散型大语言模型（dLLMs）是自回归大语言模型（AR-LLMs）的有前景替代方案，因其可能带来更高的推理吞吐量。强化学习（RL）对使dLLMs在推理等关键任务上达到与AR-LLMs相当的性能至关重要。然而，针对dLLMs特性的合适RL算法尚未开发。本文提出分布匹配策略优化（DMPO），一种理论严谨的RL微调方法，通过交叉熵优化将dLLM策略分布匹配到最优、奖励倾斜分布，以增强其推理能力。我们识别出小批量训练中的关键挑战，并提出基于新型权重基线减法技术的有效解决方案。DMPO在多个推理基准上表现出色，无需监督微调，相较于先前非DMPO RL基线最高提升39.63个百分点，比基线模型高出67.97个百分点，验证了分布匹配框架的有效性。代码已公开于https://github.com/yuchen-zhu-zyc/DMPO。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) are promising alternatives to autoregressive large language models (AR-LLMs), as they potentially allow higher inference throughput. Reinforcement learning (RL) is crucial to enabling dLLMs to achieve performance comparable to that of AR-LLMs on important tasks, such as reasoning. However, RL algorithms well-suited to dLLMs' unique characteristics have yet to be developed. This paper proposes Distribution Matching Policy Optimization (DMPO), a principled and theoretically grounded RL fine-tuning method specifically designed to enhance the reasoning capabilities of dLLMs by matching the dLLM policy distribution to the optimal, reward-tilted one through cross-entropy optimization. We identify a key implementation challenge with small training batch sizes and propose several effective solutions based on a novel weight baseline subtraction technique. DMPO exhibits superior performance on multiple reasoning benchmarks without supervised fine-tuning, achieving up to a $39.63$ percentage-point improvement in accuracy over prior non-DMPO RL baselines and $67.97$ percentage points over the base model, underscoring the effectiveness of the distribution-matching framework. Our code is available at https://github.com/yuchen-zhu-zyc/DMPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。