让扩散模型生成文本时按语义边界动态分块，提升效率与质量。
SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs
- 基于语义边界预测构建动态分块，取代固定大小或符号分割。
- 在GSM8K、MATH等数据集上优于固定分块和AdaBlock方法。
- 适用于需要高效生成且保持语义连贯性的场景，如代码与数学推理。
扩散语言模型（DLM）通过迭代去噪生成文本，块式解码通过局部提交令牌提升了实用性。然而，现有方法通常依赖固定块大小或基于分隔符的运行时信号，未必与语义边界对齐。本文提出SemBlock，一种面向扩散语言模型的语义边界驱动动态块解码框架。该方法将动态块构建建模为语义边界预测，并在冻结的LLaDA隐藏状态上训练轻量级预测器。为提供监督信号，我们构建了SemBound数据集，其边界标签来源于自然语言、数学与代码任务中的话语单元、推理步骤及实现片段。推理时，SemBlock利用预测的边界概率确定每个动态块的结束位置。在GSM8K、IFEval、MATH和HumanEval上的实验表明，SemBlock持续优于固定块解码和AdaBlock方法。代码已公开：https://github.com/TH-AI-Lab-PKU/SemBlock。
原文摘要 · Abstract (English)
Diffusion language models (DLMs) generate text through iterative denoising, and blockwise decoding improves their practicality by committing tokens in local blocks. However, existing blockwise methods typically rely on fixed block sizes or delimiter-based runtime signals, which do not necessarily align with semantic boundaries. In this paper, we propose SemBlock, a semantic-boundary-driven dynamic block decoding framework for diffusion LLMs. SemBlock formulates dynamic block construction as semantic boundary prediction and trains lightweight predictors on frozen LLaDA hidden states. To provide supervision, we construct SemBound, a semantic-boundary dataset that derives boundary labels from discourse units, reasoning steps, and implementation spans across natural language, math, and code tasks. During inference, SemBlock uses predicted boundary probabilities to select the ending position of each dynamic block. Experiments on GSM8K, IFEval, MATH, and HumanEval show that SemBlock consistently improves over fixed-block decoding and AdaBlock. Our code is publicly available: https://github.com/TH-AI-Lab-PKU/SemBlock.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。