让扩散语言模型上下文扩展至128K，无需重新训练
UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
- 改进旋转位置编码，适配扩散过程的概率建模
- 实测128K上下文任务性能显著优于无需训练的基线
- 为高效实现超长上下文提供可复用的技术方案
扩散语言模型（Diffusion LLMs）潜力巨大，但其长上下文行为仍不明确。本文针对无需从头训练的后训练技术，研究如何扩展扩散语言模型（即LLaDA）的上下文窗口。通过改进标准旋转位置编码（RoPE）的扩展方式，有效适应扩散过程的内在概率建模，实现稳定长上下文扩展。进一步比较了后训练中的掩码策略，分析其对优化稳定性和长程召回的影响。基于这些发现，我们提出UltraLLaDA，一个具备128K-token上下文窗口的扩散语言模型。在长上下文任务上的实证评估显示，其性能显著优于无需训练的基线。实验结果表明，特殊的位置编码扩展是实现扩散模型长上下文扩展的关键，为实践者提供了高效实现128K级上下文的实用指导。
原文摘要 · Abstract (English)
Diffusion LLMs have attracted growing interest, with plenty of recent work emphasizing their great potential in various downstream tasks; yet the long-context behavior of diffusion LLMs remains largely uncharted. We present a case study of post-training techniques for extending the context window of diffusion LLMs (i.e., LLaDA) without retraining from scratch. We show that a simple modification to the standard Rotary Positional Embeddings (RoPE) extension effectively accommodates the probabilistic modeling inherent in the diffusion process, enabling stable scaling to longer context ranges. We further compare masking strategies used during post-training and analyze their impact on optimization stability and long-range recall. Instantiating these insights, we introduce UltraLLaDA, a diffusion LLM with a 128K-token context window that, in our empirical evaluation on long-context tasks, significantly outperforms training-free baselines. Our experimental results highlight the special positional extension as a key lever for scaling diffusion LLMs to extended contexts and offer practical guidance for practitioners seeking 128K-scale context via efficient post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。