不跳过细节的图像生成加速方法,1秒生成1024×1024图像
SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration
- 基于注意力稀疏性动态预测高分辨率稀疏模式
- 1秒完成8B模型生成1024×1024图像,速度超FlashAttention 5倍
- 适合追求高速且高质量图像生成的开发者
视觉自回归(VAR)建模因其新颖的逐尺度预测范式受到关注。然而主流方法在每一步自回归中对所有历史尺度的令牌进行全量注意力计算,导致计算复杂度随分辨率呈四次方增长,造成显著延迟。以往加速方法常跳过高分辨率尺度,虽提升速度但损失高频细节,影响图像质量。为此,我们提出训练无关的SparVAR加速框架,利用VAR注意力的三个特性:强注意力聚集、跨尺度激活相似性与明显局部性。通过从稀疏决策尺度动态预测后续高分辨率稀疏注意力模式,并设计高效索引映射机制构建尺度自相似稀疏注意力,实现大尺度下的高效稀疏计算。进一步提出跨尺度局部稀疏注意力与块级稀疏核,前向速度超过FlashAttention 5倍。大量实验表明,SparVAR可将8B模型生成1024×1024图像的时间降至1秒,且不跳过最后尺度;相比使用FlashAttention加速的基线,提速1.57倍并几乎保留全部高频细节。结合现有跳尺度策略,最高可达2.28倍加速,同时保持优异视觉生成质量。代码已公开。
原文摘要 · Abstract (English)
Visual AutoRegressive (VAR) modeling has garnered significant attention for its innovative next-scale prediction paradigm. However, mainstream VAR paradigms attend to all tokens across historical scales at each autoregressive step. As the next scale resolution grows, the computational complexity of attention increases quartically with resolution, causing substantial latency. Prior accelerations often skip high-resolution scales, which speeds up inference but discards high-frequency details and harms image quality. To address these problems, we present \textbf{SparVAR}, a training-free acceleration framework that exploits three properties of VAR attention: \textbf{(i) strong attention sinks}, \textbf{(ii) cross-scale activation similarity}, and \textbf{(iii) pronounced locality}. Specifically, we dynamically predict the sparse attention pattern of later high-resolution scales from a sparse decision scale, and construct scale self-similar sparse attention via an efficient index-mapping mechanism, enabling high-efficiency sparse attention computation at large scales. Furthermore, we propose cross-scale local sparse attention and implement an efficient block-wise sparse kernel, which achieves $\mathbf{> 5\times}$ faster forward speed than FlashAttention. Extensive experiments demonstrate that the proposed SparVAR can reduce the generation time of an 8B model producing $1024\times1024$ high-resolution images to the \textbf{1s}, \textbf{without skipping the last scales}. Compared with the VAR baseline accelerated by FlashAttention, our method achieves a $\mathbf{1.57\times}$ speed-up while preserving almost all high-frequency details. When combined with existing scale-skipping strategies, SparVAR attains up to a $\mathbf{2.28\times}$ acceleration, while maintaining competitive visual generation quality. Code is available at \href{https://github.com/CAS-CLab/SparVAR}{SparVAR}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。