无需计算似然即可优化扩散语言模型，提升生成质量与速度
LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models
- 将向量场流动匹配映射到离散标记空间，通过对比更新直接优化去噪逻辑
- 在代码与推理基准上超越现有最佳方法，推理步数减少约20%
- 适合需要高准确率的生成任务，如数学推理与代码生成
基于可验证奖励的强化学习（RLVR）在提升自回归模型性能方面取得显著进展，尤其在数学推理和代码生成等需正确性的领域。然而，直接将此类范式应用于扩散大语言模型（dLLMs）时，因精确似然计算不可行，现有方法被迫依赖高方差近似，导致性能受限。为此，我们提出无似然策略优化（LFPO），将向量场流匹配概念映射至离散标记空间。LFPO将对齐建模为几何速度修正，通过对比更新直接优化去噪逻辑，有效规避似然近似带来的误差，实现精准梯度估计。此外，LFPO通过从中间步骤预测最终解来强制一致性，有效拉直概率流，从而在显著减少迭代次数的情况下实现高质量生成。大量实验表明，LFPO不仅在代码与推理基准上优于当前最优基线，且通过减少扩散步数使推理速度提升约20%。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has achieved remarkable success in improving autoregressive models, especially in domains requiring correctness like mathematical reasoning and code generation. However, directly applying such paradigms to Diffusion Large Language Models (dLLMs) is fundamentally hindered by the intractability of exact likelihood computation, which forces existing methods to rely on high-variance approximations. To bridge this gap, we propose Likelihood-Free Policy Optimization (LFPO), a native framework that maps the concept of vector field flow matching to the discrete token space. Specifically, LFPO formulates alignment as geometric velocity rectification, which directly optimizes denoising logits via contrastive updates. This design effectively bypasses the errors inherent in likelihood approximation, yielding the precise gradient estimation. Furthermore, LFPO enforce consistency by predicting final solutions from intermediate steps, effectively straightening the probability flow to enable high-quality generation with significantly fewer iterations. Extensive experiments demonstrate that LFPO not only outperforms state-of-the-art baselines on code and reasoning benchmarks but also accelerates inference by approximately 20% through reduced diffusion steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。