arXiv:2605.15676cs.CL2026-05

用内容分块替代固定位置分块,提升扩散语言模型的生成效果

Dynamic Chunking for Diffusion Language Models

论文配图:Dynamic Chunking for Diffusion Language Models
图 1 · 摘自论文原文
  • 基于语义内容动态分块,取代固定位置分块
  • 在1.5B参数下,优于无结构和位置分块基线
  • 适合追求高效序列建模与生成质量的研究者

块式离散扩散语言模型将序列按固定位置块自回归分解,解耦块内并行去噪与块间条件依赖。我们指出这种刚性划分浪费了序列中已有的结构:按位置定义的块会割裂语义连贯的词元,混淆无关词元。为此提出动态分块扩散模型(DCDM),以内容定义的语义块替代位置块。核心是可微的分块注意力层,通过可学习子空间将词元路由至K个聚类,并由扩散目标端到端优化。由此产生的聚类分配生成块因果注意力掩码,使离散扩散去噪器在语义块上自回归分解序列似然,严格泛化块式扩散模型。在参数规模达1.5B的下游任务中,DCDM持续优于无结构及位置块扩散基线,优势在不同规模下稳定且训练初期即显现。

原文摘要 · Abstract (English)

Block discrete diffusion language models factorize a sequence autoregressively over fixed-size positional blocks, decoupling within-block parallel denoising from across-block conditioning. We argue that this rigid partition wastes structure already present in the sequence: blocks defined by position rather than by content separate semantically coherent tokens and group unrelated ones together. We introduce the \textbf{D}ynamic \textbf{C}hunking \textbf{D}iffusion \textbf{M}odel (DCDM), which replaces positional blocks with content-defined semantic chunks. At its core is Chunking Attention, a differentiable layer that routes tokens into $K$ clusters parameterized by learnable subspaces and shaped end-to-end by the diffusion objective. The resulting cluster assignments induce a chunk-causal attention mask under which a discrete diffusion denoiser factorizes the sequence likelihood autoregressively over semantic chunks, strictly generalizing block discrete diffusion. On downstream benchmarks at parameter scales up to 1.5B, DCDM consistently improves over both unstructured and positional-block diffusion baselines, with the advantage stable across scales and visible early in training.

扩散模型语言建模动态分块

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。