提出高效微调框架DiRL，让扩散语言模型数学推理更强
DiRL: An Efficient Post-Training Framework for Diffusion Language Models
- 用块级训练+推理优化实现快速在线更新
- 在数学数据上超越同类扩散模型和同尺寸Qwen系列
- 首个适配扩散模型的无偏强化学习算法
扩散语言模型（dLLMs）作为自回归模型的有前途替代方案，虽已验证预训练潜力并加速推理，但其后训练体系仍不完善。现有方法存在计算效率低和训练-推理目标不匹配问题，严重制约其在数学等复杂推理任务上的表现。为此，我们提出DiRL，一种高效后训练框架，通过集成FlexAttention加速的块级训练与LMDeploy优化的推理，构建流畅的在线模型更新机制，支持高效的两阶段后训练（监督微调+强化学习）。基于此框架，我们提出DiPO，首个专为dLLMs设计的无偏组相对策略优化实现。我们在高质量数学数据上训练了DiRL-8B-Instruct模型，结果表明该模型在dLLMs中达到最优数学性能，并在多个基准测试中超越同规模Qwen2.5系列模型。
原文摘要 · Abstract (English)
Diffusion Language Models (dLLMs) have emerged as promising alternatives to Auto-Regressive (AR) models. While recent efforts have validated their pre-training potential and accelerated inference speeds, the post-training landscape for dLLMs remains underdeveloped. Existing methods suffer from computational inefficiency and objective mismatches between training and inference, severely limiting performance on complex reasoning tasks such as mathematics. To address this, we introduce DiRL, an efficient post-training framework that tightly integrates FlexAttention-accelerated blockwise training with LMDeploy-optimized inference. This architecture enables a streamlined online model update loop, facilitating efficient two-stage post-training (Supervised Fine-Tuning followed by Reinforcement Learning). Building on this framework, we propose DiPO, the first unbiased Group Relative Policy Optimization (GRPO) implementation tailored for dLLMs. We validate our approach by training DiRL-8B-Instruct on high-quality math data. Our model achieves state-of-the-art math performance among dLLMs and surpasses comparable models in the Qwen2.5 series on several benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。