不切分轨迹的强化学习方法,让扩散语言模型训练更高效。
SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicing

- 用风险预算解码器控制每步生成风险,避免重构推理轨迹。
- 在SDAR-4B上仅用0.46倍样本量达到TraceRL最佳性能,数学题准确率提升6.32%。
- 适用于想提升训练效率且关注数学/代码生成的模型开发者。
扩散大语言模型(dLLMs)的强化学习正转向轨迹感知方法。当前最优方法TraceRL认为随机掩码与模型推理轨迹不匹配,需将每个轨迹切分为最多K/s个对齐样本,成本随块大小K增加。本文提出SLIM-RL,无需重构轨迹即可缓解此不匹配:通过tau预算解码器约束每步提交风险,降低训练数据整体风险。优化时,使用无轨迹依赖的随机掩码目标,结合序列级重要性采样与确定性求积,采用均值保持、单调递减的每块掩码调度策略。在SDAR-4B上,SLIM-RL以0.46倍样本量达到TraceRL最佳MATH500准确率,动态采样条件下比TraceRL提升6.32%(MATH500)和11.05%(GSM8K)。块大小为4时,4B版SLIM-RL超越更大的LLaDA-8B和Dream-7B,在数学任务上优于后者10.76%(MATH500),且低于自回归Qwen2.5-7B。在代码任务上,比TraceRL提升4.20%(MBPP)和3.65%(HumanEval)。tau预算解码器可无训练迁移至LLaDA、Dream和SDAR。源码见https://github.com/laolaorkkkkk/SLIM-RL。
原文摘要 · Abstract (English)
Reinforcement learning for diffusion large language models (dLLMs) has largely moved to trajectory-aware methods. The current state of the art, TraceRL, holds that random masking is mismatched with the model's inference trajectory, and it reconstructs that trajectory during training by slicing each rollout into up to K/s trajectory-aligned training samples, a cost that grows with the block size K. We show that this mismatch can be mitigated without reconstructing the trajectory. Our method, SLIM-RL, bounds the commit risk of each rollout step with a tau-budget decoder, reducing aggregate commit risk in the training data. During optimization, SLIM-RL trains on these risk-controlled rollouts with a trace-free random-masking objective that adapts variance-reduction tools, combining sequence-level importance sampling, deterministic quadrature over masking levels under a mean-preserving, monotonically decreasing per-block mask schedule that we introduce. On SDAR-4B, SLIM-RL matches TraceRL's best MATH500 accuracy on only 0.46x its training samples at block size 16, improving over TraceRL by 6.32% on MATH500 and 11.05% on GSM8K under matched dynamic sampling. At block size 4, the 4B SLIM-RL surpasses the larger LLaDA-8B and Dream-7B dLLMs on math, exceeding LLaDA-8B by 10.76% on MATH500 while staying below the autoregressive Qwen2.5-7B. On code, it improves over TraceRL by 4.20% on MBPP and 3.65% on HumanEval. The tau-budget decoder transfers training-free across LLaDA, Dream, and SDAR. The source code is available at https://github.com/laolaorkkkkk/SLIM-RL .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。