动态稀疏注意力让图像生成更快更省显存。
Make It Efficient: Dynamic Sparse Attention for Autoregressive Image Generation
- 根据生成阶段动态筛选关键历史标记，只关注必要信息。
- 推理时显存减少约50%，生成质量无损失。
- 无需重新训练，适合部署在资源受限的场景。
自回归图像生成模型已成为文本到图像合成的主流方法。这类方法将图像转为一维标记序列，并利用在自然语言处理中表现卓越的自注意力机制来捕捉长程依赖、建模全局上下文并保证语义连贯性。然而，推理时过长的上下文导致键值缓存（KV-cache）占用大量显存并引发计算延迟。为此，我们系统分析了生成过程中全局语义、空间布局与细粒度纹理的形成机制，提出一种无需训练的上下文优化方法——自适应动态稀疏注意力（ADSA）。ADSA可动态识别维持局部纹理一致性与全局语义连贯性所必需的历史标记，从而高效简化注意力计算。此外，我们设计了适配ADSA的动态KV缓存更新机制，使推理期间GPU显存消耗降低约50%。大量定性和定量实验表明，该方法在生成质量与资源效率方面均表现出色。
原文摘要 · Abstract (English)
Autoregressive conditional image generation models have emerged as a dominant paradigm in text-to-image synthesis. These methods typically convert images into one-dimensional token sequences and leverage the self-attention mechanism, which has achieved remarkable success in natural language processing, to capture long-range dependencies, model global context, and ensure semantic coherence. However, excessively long contexts during inference lead to significant memory overhead caused by KV-cache and computational delays. To alleviate these challenges, we systematically analyze how global semantics, spatial layouts, and fine-grained textures are formed during inference, and propose a novel training-free context optimization method called Adaptive Dynamic Sparse Attention (ADSA). Conceptually, ADSA dynamically identifies historical tokens crucial for maintaining local texture consistency and those essential for ensuring global semantic coherence, thereby efficiently streamlining attention computation. Additionally, we introduce a dynamic KV-cache update mechanism tailored for ADSA, reducing GPU memory consumption during inference by approximately $50\%$. Extensive qualitative and quantitative experiments demonstrate the effectiveness and superiority of our approach in terms of both generation quality and resource efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。