用稀疏注意力提升长文本生成效率，降低计算开销。
SA-DiffuSeq: Addressing Computational and Scalability Challenges in Long-Document Generation with Sparse Attention
- 引入稀疏注意力机制，仅在关键位置计算注意力。
- 在长序列上生成速度提升显著，保持语义连贯性。
- 适合科学写作、代码生成等长文本场景。
基于扩散模型的长文本生成面临序列长度增加导致的计算成本过高和内存开销大问题。本文提出SA-DiffuSeq，通过在扩散过程中引入稀疏注意力机制，从根本上提升长文档建模的可扩展性。该方法通过选择性分配注意力，显著降低计算复杂度，同时保持语义连贯性和生成质量。核心设计包括一种针对稀疏注意力动态的软吸收态，稳定扩散轨迹并加速序列重建，从而提升采样效率，增强长距离依赖建模精度。大量实验表明，SA-DiffuSeq在训练效率和采样速度上持续优于现有扩散基基线，尤其在长序列任务中表现突出。该模型适用于科学写作、大规模代码生成及多轮长上下文对话等高要求场景。结果表明，将结构化稀疏性融入扩散模型是实现高效且富有表现力的长文本生成的有前景方向。
原文摘要 · Abstract (English)
Diffusion based approaches to long form text generation suffer from prohibitive computational cost and memory overhead as sequence length increases. We introduce SA-DiffuSeq, a diffusion framework that integrates sparse attention to fundamentally improve scalability for long document modeling. By selectively allocating attention within the diffusion process, SA-DiffuSeq significantly reduces computational complexity while maintaining semantic coherence and generation quality. A key component of our method is a soft absorbing state tailored to sparse attention dynamics, which stabilizes diffusion trajectories and accelerates sequence reconstruction. This design improves sampling efficiency and enhances precision in long range dependency modeling. Extensive experiments demonstrate that SA-DiffuSeq consistently surpasses state of the art diffusion baselines in both training efficiency and sampling speed, with especially strong gains on extended sequences. These properties make SA-DiffuSeq well suited for demanding long form applications such as scientific writing, large scale code generation, and multi turn long context dialogue. Overall, our results indicate that incorporating structured sparsity into diffusion models is a promising direction for efficient and expressive long text generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。