加一个特殊标记稳定扩散语言模型，提升生成质量。
One Token Is Enough: Improving Diffusion Language Models with a Sink Token
- 引入仅自关注的特殊令牌作为结构化锚点。
- 单个额外令牌显著改善注意力聚焦稳定性。
- 无需语义内容，位置无关，适合部署优化。
扩散语言模型（DLMs）作为自回归方法的有力替代，实现了并行文本生成且性能相当。然而，其存在关键不稳定性：注意力焦点漂移现象。分析表明，下沉令牌在Transformer的值空间中具有低范数特征，而漂移现象实为防止信息过度混合的保护机制。但其在扩散步骤间位置不可预测，损害推理鲁棒性。为此，我们提出一种简单有效的改进方案：通过修改注意力掩码引入一个额外的特殊令牌。该令牌被约束仅自关注，同时对所有其他令牌全局可见。实验表明，引入单个额外令牌即可稳定注意力下沉，显著提升模型性能。进一步分析证实，该令牌的有效性与其位置无关，且语义含量可忽略，验证了其作为稳健专用结构锚点的角色。
原文摘要 · Abstract (English)
Diffusion Language Models (DLMs) have emerged as a compelling alternative to autoregressive approaches, enabling parallel text generation with competitive performance. Despite these advantages, there is a critical instability in DLMs: the moving sink phenomenon. Our analysis indicates that sink tokens exhibit low-norm representations in the Transformer's value space, and that the moving sink phenomenon serves as a protective mechanism in DLMs to prevent excessive information mixing. However, their unpredictable positions across diffusion steps undermine inference robustness. To resolve this, we propose a simple but effective extra sink token implemented via a modified attention mask. Specifically, we introduce a special token constrained to attend solely to itself, while remaining globally visible to all other tokens. Experimental results demonstrate that introducing a single extra token stabilizes attention sinks, substantially improving model performance. Crucially, further analysis confirms that the effectiveness of this token is independent of its position and characterized by negligible semantic content, validating its role as a robust and dedicated structural sink.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。