让大模型推理块大小自适应，提升逻辑连贯性。
Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning
- 用强化学习优化熵下降趋势，动态调整推理块大小。
- 在多个推理任务上显著提升一致性，优于固定块大小方法。
- 可无缝接入现有模型，适合追求推理质量的研究者。
最近的扩散大语言模型（dLLMs）通过基于块的半自回归生成范式，在推理中展现出高效与有效。然而，固定大小的块生成仍是影响推理效果与连贯性的关键瓶颈。一方面，不同推理任务应对应不同最优解码块大小，单一尺寸无法适配；另一方面，僵化的块划分会破坏逻辑流程，降低连贯性。通过实证观察发现，错误推理的块级熵呈现波动不稳趋势，而正确推理则保持持续下降。为此，本文提出 b1——一种新型后训练框架，通过单调熵下降目标结合强化学习，实现 dLLMs 的动态推理块大小学习。b1 可作为即插即用模块集成于现有 dLLM 后训练流程。在多个推理基准上的大量实验表明，b1 在一致性上持续优于固定块大小基线。代码已公开于 https://github.com/YanJiangJerry/Block-R1。
原文摘要 · Abstract (English)
Recent diffusion large language models (dLLMs) have demonstrated both effectiveness and efficiency in reasoning via a block-based semi-autoregressive generation paradigm. Despite their progress, the fixed-size block generations remain a critical bottleneck for effective and coherent reasoning. 1. From a global perspective, different reasoning tasks would correspond to different optimal decoding block sizes, which makes a ``one-size-fits-all'' assumption ineffective. 2. Even within a single reasoning task, the rigid block partitioning would break the logical flow and reduce reasoning coherence. Through empirical observations, we reveal that for block-wise entropy, incorrect reasoning exhibits a fluctuating and unsteady trend between blocks, whereas the correctly generated tasks follow a consistent descending trend. Therefore, this paper proposes b1, a novel post-training framework for dLLMs that learns dynamic-size reasoning blocks via a Monotonic Entropy Descent objective with reinforcement learning to enhance reasoning coherence.b1 integrates seamlessly as a plug-and-play module with existing dLLM's post-training algorithms. Extensive experiments across various reasoning benchmarks showcase b1's consistent improvement over existing fixed-size block baselines. Our code has been released at https://github.com/YanJiangJerry/Block-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。