让扩散语言模型更准地生成文本，关键在先锚定重要词
Anchored Diffusion Language Model
- 先用锚定网络预测关键词分布，再基于此重建缺失文本
- 在LM1B和OpenWebText上困惑度降低25.4%，接近自回归模型
- 首次实现扩散模型生成文本比自回归模型更像人写的
扩散语言模型（DLMs）虽具并行生成与双向上下文优势，但在似然建模与文本质量上仍逊于自回归（AR）模型。我们发现，问题源于关键词（如关键词或低频词）在前向过程早期被掩码，导致重构时上下文信息不足。为此，提出锚定扩散语言模型（ADLM），一种两阶段框架：首先通过锚定网络预测重要词的分布，再基于锚定结果预测缺失词的似然。ADLM在LM1B和OpenWebText上测试困惑度显著降低，相比以往DLM最高提升25.4%，大幅缩小与强自回归基线的差距。在七个零样本基准上达顶尖性能，且在MAUVE评分上超越自回归模型，首次实现扩散模型生成文本优于自回归模型。理论上，我们推导出锚定负证据下界（ANELBO）目标，证明锚定可提升采样复杂度与似然建模能力。该思想亦适用于自回归模型，增强数学与逻辑推理表现，优于现有思维链方法。
原文摘要 · Abstract (English)
Diffusion Language Models (DLMs) promise parallel generation and bidirectional context, yet they underperform autoregressive (AR) models in both likelihood modeling and generated text quality. We identify that this performance gap arises when important tokens (e.g., key words or low-frequency words that anchor a sentence) are masked early in the forward process, limiting contextual information for accurate reconstruction. To address this, we introduce the Anchored Diffusion Language Model (ADLM), a novel two-stage framework that first predicts distributions over important tokens via an anchor network, and then predicts the likelihoods of missing tokens conditioned on the anchored predictions. ADLM significantly improves test perplexity on LM1B and OpenWebText, achieving up to 25.4% gains over prior DLMs, and narrows the gap with strong AR baselines. It also achieves state-of-the-art performance in zero-shot generalization across seven benchmarks and surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model. Theoretically, we derive an Anchored Negative Evidence Lower Bound (ANELBO) objective and show that anchoring improves sample complexity and likelihood modeling. Beyond diffusion, anchoring boosts performance in AR models and enhances reasoning in math and logic tasks, outperforming existing chain-of-thought approaches
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。