arXiv:2506.16054cs.CVcs.GR2025-06NeurIPS被引 17

通过重排注意力模式,实现高效低精度视觉生成。

PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

  • 提出PARO方法,将杂乱注意力重排为块状结构。
  • 在INT8/INT4下保持无损性能,延迟降低1.9至2.7倍。
  • 适合追求高效推理的图像与视频生成应用。

在视觉生成中,注意力机制的二次复杂度导致高内存和计算开销,尤其在高分辨率图像或多帧视频生成所需的长序列下更为显著。现有研究尝试稀疏化和量化以缓解问题,但在低密度和低比特位宽下仍面临挑战。我们系统分析发现,根源在于视觉注意力模式的分散与不规则性。因此,我们提出一种新策略:重新组织注意力模式,而非设计专用稀疏化与量化方案。受视觉特征提取的局部聚集特性启发,我们设计了新型**模式感知标记重排(PARO)** 技术,将多样化注意力模式统一为硬件友好的块状结构,显著简化并提升稀疏化与量化效果。我们评估了多种设计选择,最终确定适配统一模式的方法。所提出的**PAROAttention**在视频与图像生成中实现无损指标,结果几乎与全精度(FP)基线一致,同时运行密度仅约20%-30%,比特位宽降至INT8/INT4,端到端延迟提速1.9至2.7倍。

原文摘要 · Abstract (English)

In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolution image or multi-frame video generation. To address this, prior research has explored techniques such as sparsification and quantization. However, these techniques face significant challenges under low density and reduced bitwidths. Through systematic analysis, we identify that the core difficulty stems from the dispersed and irregular characteristics of visual attention patterns. Therefore, instead of introducing specialized sparsification and quantization design to accommodate such patterns, we propose an alternative strategy: *reorganizing* the attention pattern to alleviate the challenges. Inspired by the local aggregation nature of visual feature extraction, we design a novel **Pattern-Aware token ReOrdering (PARO)** technique, which unifies the diverse attention patterns into a hardware-friendly block-wise pattern. This unification substantially simplifies and enhances both sparsification and quantization. We evaluate the performance-efficiency trade-offs of various design choices and finalize a methodology tailored for the unified pattern. Our approach, **PAROAttention**, achieves video and image generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating at notably lower density (~20%-30%) and bitwidth (**INT8/INT4**), achieving a **1.9x** to **2.7x** end-to-end latency speedup.

注意力机制稀疏化量化视觉生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。