让扩散语言模型的推理更准：通过精准奖励与真实生成路径对齐。
Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
- 用分步奖励机制把终点奖励拆成每一步的精细反馈。
- 在高不确定步骤重演真实生成路径，提升梯度利用效率。
- 适合研究扩散模型推理优化或强化学习应用的学者。
强化学习（RL）在提升扩散大语言模型（dLLMs）推理能力方面潜力巨大，但进展受限于双重错位：(i) 过程-奖励错位。稀疏的终局奖励被均摊给生成过程的所有中间步骤，无法实现差异化的信用分配；(ii) 状态-轨迹错位。策略更新常偏离真实生成轨迹，导致梯度浪费在低信息量样本上。为此，我们提出过程对齐策略优化（PAPO），通过步骤感知的过程奖励（SPR）将稀疏终局奖励转化为密集的步骤级信用，并采用熵引导的历史重演（EHR）在高不确定性步骤重播真实生成轨迹，从而实现与生成过程的整体对齐。在四个基准测试上的大量实验表明，PAPO显著优于基线方法，在GSM8K上提升4.5%，MATH500上提升4.8%，Countdown上提升42.2%，Sudoku上提升16.1%。
原文摘要 · Abstract (English)
Reinforcement learning (RL) holds immense promise for enhancing the reasoning capabilities of diffusion large language models (dLLMs). However, progress is fundamentally constrained by a dual misalignment between authentic generation trajectory and the gradient update process: (i) Process-reward misalignment. Sparse, terminal rewards are indiscriminately assigned to all intermediate steps of the generation process, failing to provide discriminative credit assignment. (ii) State-trajectory misalignment. Policy updates are often diverted toward artificial, out-of-trajectory states, squandering gradients on less informative samples. To address these limitations, we introduce Process Aligned Policy Optimization (PAPO), a novel framework that holistically aligns the RL update with the dLLM's generative trajectory via Step-Aware Process Rewards (SPR) that transform sparse terminal rewards into dense, step-wise credit, and Entropy-Guided Historical Re-enactment (EHR) that replays authentic trajectories at high-uncertainty steps. Extensive experiments on four benchmarks demonstrate that PAPO significantly outperforms baselines, achieving gains of up to 4.5% on GSM8K, 4.8% on MATH500, 42.2% on Countdown and 16.1% on Sudoku.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。