用[VOID]替代[EOS]做填充，让模型更准停顿、更快生成。
VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination
![论文配图：VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination](https://arxiv.org/html/2606.17999v2/figures/void_padding_overview.png)
- 引入[VOID]专门处理填充，让[EOS]只负责语义终止
- 在Dream-7B上提升17.84分，解码计算量减少55.7%
- 适合追求高效生成与精准结束的推理场景
MDLM通过去噪预设的掩码响应画布生成文本，响应长度建模是指令微调的核心。现有模型继承自回归惯例，在指令微调中使用重复的[EOS]作为填充，使[EOS]同时承担语义终止和填充双重角色。我们发现这是大块解码时[EOS]溢出的根本原因。为此提出VoidPadding，引入[VOID]处理填充，保留[EOS]用于终止。推理时，学习到的[EOS]信号实现早期停止，[VOID]信号引导自适应响应画布扩展。在Dream-7B-Instruct上，相较于原模型提升17.84分，较RainbowPadding提升6.95分，平均解码非线性求解器调用次数（NFE）降低55.7%。代码已开源。
原文摘要 · Abstract (English)
MDLMs generate text by denoising a preallocated masked response canvas, making response-length modeling central to instruction tuning. Existing MDLMs often inherit the autoregressive convention of using repeated \texttt{[EOS]} tokens for padding during instruction tuning, giving \texttt{[EOS]} a dual role as both a semantic terminator and a padding token. We show that this dual role is a root cause of \texttt{[EOS]} overflow under large-block decoding. To decouple these roles, we propose VoidPadding, which introduces \texttt{[VOID]} for padding and reserves \texttt{[EOS]} for termination. During inference, the learned \texttt{[EOS]} signal enables early stopping, while the learned \texttt{[VOID]} signal guides adaptive response canvas expansion. On Dream-7B-Instruct, VoidPadding improves the block-size-averaged four-task mean across mathematical reasoning and code generation benchmarks by \(+17.84\) points over the original model and \(+6.95\) points over RainbowPadding, while reducing decoding NFE by 55.7\% on average. Code is available at https://github.com/Haru-LCY/VoidPadding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。