改进扩散语言模型的强化学习，提升生成质量与一致性。
DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
- 引入去噪进度评分，动态分配每个步骤的重要性权重。
- 通过分层掩码降低均值场偏差，显著提升策略优化精度。
- 轻量级插件设计，适用于多种强化学习框架，效果稳定提升。
扩散大语言模型是自回归模型的有力替代，但现有强化学习方法将所有去噪步骤视为同等重要，并依赖有偏且高方差的似然估计。我们识别出两个根本问题：去噪轨迹中缺乏时间上的信用分配，以及用于策略优化的均值场似然估计存在系统性偏差。为此，我们提出去噪感知信用分配的GRPO（DACA-GRPO），一种轻量级、可即插即用的GRPO类训练器增强方法。DACA-GRPO引入两种互补机制：去噪进度评分，从中间预测中无额外前向开销提取每个词元的重要性权重；分层掩码似然，将词元位置划分为若干层级，使每个词元在大部分序列上下文中进行预测，从而降低均值场偏差。在三种GRPO基础方法上应用，DACA-GRPO在七个基准上实现一致提升，涵盖数学推理、代码生成、约束满足和受限生成任务，数学推理最高提升5.6个百分点，代码生成提升7.4个百分点，约束满足提升36.3个百分点，JSON模式遵循度提升5.9个百分点。
原文摘要 · Abstract (English)
Diffusion large language models are a compelling alternative to autoregressive models, yet existing RL methods for diffusion treat all denoising steps as equally important and rely on biased, high-variance likelihood estimates. We identify two fundamental weaknesses: the absence of temporal credit assignment across the denoising trajectory, and the systematic bias of mean-field likelihood estimates used for policy optimization. To address these, we propose Denoising-Aware Credit Assignment for GRPO (DACA-GRPO), a lightweight, plug-and-play enhancement for any GRPO-style trainer. DACA-GRPO introduces two complementary mechanisms: Denoising Progress Scores, which extract per-token importance weights from intermediate predictions at no additional forward cost, and Stratified Masking Likelihood, which partitions token positions into strata so that each token is predicted with most of the sequence as context, reducing the mean-field bias. Applied on top of three GRPO base methods, DACA-GRPO achieves consistent improvements across seven benchmarks spanning mathematical reasoning, code generation, constraint satisfaction, and constrained generation, with gains of up to 5.6pp on math reasoning, 7.4pp on code generation, 36.3pp on constraint satisfaction, and 5.9pp on JSON schema adherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。