改进扩散语言模型的预测反馈机制，提升生成质量与稳定性。
Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models
- 采用球面线性插值（SLERP）替代线性插值，更适配嵌入空间的球形几何。
- 在不同采样预算下，生成困惑度降低16.9%-19.6%，MAUVE得分最高提升2倍。
- 适合关注生成质量、训练稳定性的扩散模型研究者和实践者。
软掩码能加速掩码扩散语言模型（MDLMs）的收敛。现有方法在原始嵌入空间中使用线性插值（LERP）实现混合，隐含假设该空间为欧氏空间。我们分析了MDLM的嵌入空间，发现掩码与预测词嵌入间的夹角始终维持约73°，且嵌入范数在词汇频率排名上基本平坦，表明其具有超球面几何特性，而LERP在此不适用。为此，我们提出球面软掩码（S-SM），以超球面上的Fréchet均值聚合前k个预测结果，并通过球面线性插值（SLERP）融合掩码方向，最后恢复原掩码范数。在释放的169M参数MDLM检查点上进行持续预训练，评估了多种推理步数预算下的表现。相比原始基线，S-SM使生成困惑度降低16.9%-19.6%，MAUVE得分最高提升2倍，相较TopK/LERP提升27.5%-56.1%；同时输出熵与收敛性几乎不变。
原文摘要 · Abstract (English)
Soft-masking accelerates the convergence of Masked Diffusion Language Models (MDLMs). Existing formulations build this blend with linear interpolation (LERP) in the raw embedding space, which implicitly treats that space as Euclidean. We analyze the embedding space of MDLMs and find that the mask and predicted-token embeddings maintain a near-constant angle of (\approx 73^\circ) throughout training, while embedding norms remain essentially flat across vocabulary-frequency rank. These indicate a hyperspherical geometry, for which LERP is the wrong interpolation primitive. We introduce Spherical Soft-Masking (S-SM), a drop-in replacement that aggregates the top-(k) predictions with a Fr'echet mean on the hypersphere and blends this mean with the mask direction using spherical linear interpolation (SLERP), then restores the native mask norm. We evaluate S-SM on continued pre-training of a released 169M-parameter MDLM checkpoint across a wide range of inference-time step budgets, SLERP feedback avoids the training degradation that LERP feedback induces and delivers MAUVE gains of up to 2x over the vanilla MDLM baseline and 27.5-56.1% over TopK/LERP at various sampling budgets, alongside consistently lower generative perplexity (16.9-19.6% over the baseline), while leaving output entropy and convergence essentially unchanged.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。