提出可变大小的自洽块,让语言模型更智能地决定何时提交生成结果。
When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Models

- 用未来感知与无未来条件下的预测差异来判断是否该提交当前块
- 在多个数据集上相比固定块和启发式方法,生成质量显著提升
- 适合追求高效且高质量文本生成的研究者或开发者
离散扩散语言模型(dLLMs)支持并行词元更新与双向注意力,但实际生成常采用分块半自回归解码。这导致训练与推理不一致:训练时使用完整序列上下文去噪,而推理中受限于固定块大小,无法获取未来上下文。因此,固定或启发式分块可能导致过早提交词元,因决策缺乏未来信息影响。为此,我们提出以‘自洽性’为原则选择块边界——一个块若在有/无未来上下文条件下预测保持一致,则为自洽。基于此,我们引入可变大小自洽块(VSB),通过比较无未来(NF)与未来感知(FA)条件下的词元级预测分布差异,量化未来信息可能带来的变化,并据此动态选择块边界。理论分析表明自洽性可保证预测一致性,大量实验验证了VSB在多个数据集上优于固定块与启发式方法。
原文摘要 · Abstract (English)
Discrete diffusion language models (dLLMs) enable parallel token updates with bidirectional attention, yet practical generation typically adopts blockwise semi-autoregressive decoding. This switch creates a training-inference mismatch: training denoises with full-sequence context, while inference commits tokens within a bounded block without future context. Therefore, decoding with fixed-size or heuristic-based blocks can lead to premature token commitments, as decisions are made without full access to future context that could alter those choices. Motivated by this, we propose self-containedness as a principled criterion for block commitment. A block is self-contained if its predictions remain consistent with Future-Aware (FA) or without No-Future (NF) access to future context, reframing block boundary selection as a test of self-containedness rather than a heuristic choice. Based on this principle, we introduce Variable-size Self-contained Blocks (VSB) for dLLMs. VSB scores and selects block boundaries using the divergence between token-level predictive distributions under NF and FA conditioning, which quantifies how predictions would change if future context were revealed. We provide theoretical justification linking self-containedness to predictive consistency, and extensive experiments validate VSB's efficacy over fixed-size and heuristic blockwise decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。