研究扩散大模型强化学习中块大小对多领域冲突的影响,提出新基准与训练方法。
Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models

- 从领域冲突角度重新审视块大小作用,揭示其对强化学习效果的关键影响
- 构建包含41K样本的Block-R1数据集,量化领域间块大小冲突程度
- 提供灵活的单域与跨域强化学习基准,适合大模型后训练研究者使用
近期,强化学习(RL)被广泛应用于扩散大语言模型(dLLMs)的后训练阶段,以通过分块半自回归生成提升推理能力。块大小因此成为dLLMs中的关键因素,因其决定并行解码粒度,并影响基于滚动优化的强化学习方法(如GRPO)的轨迹。本文首次从多领域冲突视角研究块大小在dLLM强化学习后训练中的作用。主要贡献包括:(1) 提出多领域强化学习中块大小冲突的理论框架,显著影响基于滚动的强化学习后训练效果;(2) 构建新数据集Block-R1-41K,为每个样本确定最优训练块大小,并引入块大小冲突评分进行定量评估;(3) 提出新的基准Block-R1,支持单域与跨域灵活强化学习后训练;(4) 设计一种简单而有效的跨域后训练方法,采用样本级最优改进的训练块大小。在13个不同数据集、7种最新强化学习算法及多种dLLM骨干模型上进行了全面实验。基准代码开源于https://github.com/YanJiangJerry/Block-R1,数据集发布于https://huggingface.co/datasets/YanJiangJerry/Block-R1-41K。
原文摘要 · Abstract (English)
Recently, reinforcement learning (RL) has been widely applied during post-training for diffusion large language models (dLLMs) to enhance reasoning with block-wise semi-autoregressive generation. Block size has therefore become a vital factor in dLLMs, since it determines the parallel decoding granularity and affects the rollout trajectories during RL optimisation, e.g., GRPO. Instead of investigating the effect of block size during inference on individual domains, this paper studies block size from a domain conflict perspective for dLLM RL post-training in multi-domain scenarios. The main contributions are: (1) a formulation of domain block size conflict in multi-domain RL for dLLMs, which will largely affect the post-training effectiveness for rollout-based RL methods; (2) a novel dataset, Block-R1-41K is constructed with a best-improved training block size for each sample, which also induces a Block Size Conflict Score to quantitatively measure the domain conflict; (3) a new benchmark, Block-R1, for flexible RL post-training for dLLMs in both single and cross domain; and (4) a simple yet powerful cross-domain post-training method with sample-level best-improved training block sizes. Extensive experiments on 13 distinct datasets, 7 latest RL algorithms and diverse dLLM backbones are comprehensively covered in Block-R1. The benchmark is open-sourced at https://github.com/YanJiangJerry/Block-R1 with the dataset released at https://huggingface.co/datasets/YanJiangJerry/Block-R1-41K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。