让扩散语言模型学会更好推理，显著提升数学编程能力
Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models
- 引入轨迹感知强化学习，用扩散价值模型稳定训练过程
- 40亿参数模型超越70亿自回归模型，80亿版数学准确率提升51.3%
- 开源全流程工具链，支持多种架构与任务的扩散语言模型构建
我们提出TraceRL，一种面向扩散语言模型（DLMs）的轨迹感知强化学习框架，将优选推理路径融入后训练，适用于不同模型架构。通过基于扩散的价值模型提升训练稳定性，在复杂数学与编程任务上实现性能提升。该框架还可用于将块级模型扩展至更大块，增强采样灵活性。采用TraceRL，我们构建了一系列领先水平的扩散语言模型——TraDo。尽管规模小于70亿参数的自回归模型，TraDo-4B-Instruct在复杂数学推理任务中仍持续超越它们。TraDo-8B-Instruct在数学推理基准上相较Qwen2.5-7B-Instruct提升6.1%相对准确率，相较Llama3.1-8B-Instruct提升51.3%。通过课程学习，我们还首次获得长思维链扩散语言模型，在MATH500上相较Qwen2.5-7B-Instruct实现18.1%的相对准确率提升。为促进可复现研究与实际应用，我们开源了涵盖构建、训练与部署的完整框架，集成加速的KV缓存技术与推理引擎，支持数学、编程及通用任务的监督微调与强化学习方法。代码与模型：https://github.com/Gen-Verse/dLLM-RL
原文摘要 · Abstract (English)
We propose TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs) that incorporates preferred inference trajectory into post-training, and is applicable across different architectures. Equipped with a diffusion-based value model that enhances training stability, we demonstrate improved reasoning performance on complex math and coding tasks. Besides, it can also be applied to adapt block-specific models to larger blocks, which improves sampling flexibility. Employing TraceRL, we derive a series of state-of-the-art diffusion language models, namely TraDo. Although smaller than 7B-scale AR models, TraDo-4B-Instruct still consistently outperforms them across complex math reasoning tasks. TraDo-8B-Instruct achieves relative accuracy improvements of 6.1% over Qwen2.5-7B-Instruct and 51.3% over Llama3.1-8B-Instruct on mathematical reasoning benchmarks. Through curriculum learning, we also derive the first long-CoT DLM, outperforming Qwen2.5-7B-Instruct on MATH500 with an 18.1% relative accuracy gain. To facilitate reproducible research and practical applications, we release a comprehensive open-source framework for building, training, and deploying diffusion LLMs across diverse architectures. The framework integrates accelerated KV-cache techniques and inference engines for both inference and reinforcement learning, and includes implementations of various supervised fine-tuning and RL methods for mathematics, coding, and general tasks. Code and Models: https://github.com/Gen-Verse/dLLM-RL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。