用去噪反馈训练扩散语言模型，提升生成质量与泛化能力
Reinforcement Learning from Denoising Feedback
- 从中间噪声状态中提取干净状态，结合加权时间步采样优化策略
- 在多个推理基准上显著提升LLaDA和Dream模型的性能与泛化性
- 适合研究扩散语言模型与强化学习融合的开发者使用
策略损失估计是扩散语言模型（DLMs）强化学习中的核心难题。本文提出基于去噪反馈的强化学习（RLDF），利用滚动和训练过程中的反馈，实现高效准确的策略损失估计。为平衡计算效率与估计效果，RLDF优化模型以中间噪声状态中的裁剪后干净状态为目标，并采用加权时间步采样策略处理去噪过程。大量实验表明，该方法在两个代表性DLM架构LLaDA和Dream上，于多个推理基准任务中均实现一致且显著的性能提升。本工作为可扩展的扩散语言模型强化学习提供了理论基础。我们构建了名为Drift的DLM训练框架，开源地址：https://github.com/ant-research/Drift。
原文摘要 · Abstract (English)
Policy loss estimation remains a fundamental and long-standing challenge in reinforcement learning (RL) for diffusion language models (DLMs). We introduce Reinforcement Learning from Denoising Feedback (RLDF), a novel training paradigm that leverages feedback obtained from rollout and training processes to facilitate accurate and efficient policy loss estimation. To balance the trade-off between computational efficiency and estimation effectiveness, RLDF optimizes the model toward the clipped clean state from intermediate noisy states, combined with weighted timestep sampling over denoising timesteps. Extensive experiments demonstrate that RLDF achieves consistent and substantial improvements in both performance and generalizability across two representative DLM architectures, LLaDA and Dream, on multiple reasoning benchmarks. Our work lays a principled foundation for scalable reinforcement learning in diffusion language models. We build Drift, a training framework for DLMs, available at https://github.com/ant-research/Drift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。