用外部策略生成的高质量推理路径，提升扩散语言模型的强化学习效果
Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models
- 引入外部策略生成的高奖励轨迹，弥补自身推理路径不足
- 通过长度控制和分源处理奖励，解决轨迹长度差异与训练不稳问题
- 在数独等任务上显著提升性能，尤其在零样本场景下表现突出
近期针对扩散大语言模型（dLLMs）的强化学习方法多依赖目标模型自生成的在线策略轨迹。当成功轨迹稀少时，此类方法因正向奖励匮乏而进展有限。为此，本文探索融合由更强外部策略生成的高奖励轨迹，与目标模型的在线轨迹共同训练。然而，直接整合存在两个实际挑战：轨迹长度不一致，以及联合处理不同来源奖励导致的训练不稳定。为此，提出外部轨迹集成方法ERILS，通过长度控制和分源奖励处理来应对。在数独、倒计时和MATH500三个零样本任务上的实验表明，ERILS在所有任务中均提升多样本性能，其中数独提升最显著：最佳4样本准确率达98.4%，远超最强基线的40.3%。同时，在128、256、512三种生成长度下，其确定性单次生成准确率稳定保持约90%。组件分析显示，受控长度的外部轨迹优于无控版本，且分源奖励处理可避免联合处理引发的训练崩溃。结果表明，轨迹构建与奖励处理是整合外部轨迹的关键设计维度。
原文摘要 · Abstract (English)
Recent reinforcement learning methods for diffusion large language models (dLLMs) commonly rely on on-policy rollouts generated by the target dLLM itself. When successful on-policy rollouts are scarce, however, on-policy training may receive little positive reward and make only limited progress. To mitigate this problem, we explore incorporating higher-reward rollouts generated by a stronger external policy alongside on-policy rollouts from the target dLLM. However, directly incorporating these external rollouts introduces two practical challenges: differences in rollout length and instability when jointly processing rewards from on-policy and external rollouts. To address these challenges, we propose External Rollout Integration with Length Control and Source-Specific Processing (ERILS), which controls external-rollout length and processes the rewards of on-policy and external rollouts separately. Experiments on Sudoku, Countdown, and MATH500 under zero-shot evaluation show that ERILS improves multi-sample performance across all three tasks, with the largest gains on Sudoku. On Sudoku, ERILS achieves 98.4% best-of-4 completion accuracy, compared with 40.3% for the strongest baseline. ERILS also maintains approximately 90% deterministic single-completion accuracy on Sudoku across generation lengths of 128, 256, and 512 tokens. Our component analysis further shows that length-controlled external rollouts are more effective than uncontrolled external rollouts, and that source-specific reward processing avoids the training collapse observed with joint reward processing. These results show that rollout construction and reward processing are important design dimensions when integrating external rollouts into dLLM reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。