arXiv:2511.05832cs.CVcs.AI2025-11中稿 · ICLR

用希尔伯特曲线重排图像令牌,提升局部注意力效率。

Hilbert-Guided Sparse Local Attention

  • 基于希尔伯特曲线重排序图像令牌,使局部窗口更连续。
  • 相比传统方法,窗口和滑动注意力分别提速4倍和18倍。
  • 适用于高分辨率图像任务,适合追求推理加速的开发者。

全局自注意力在高分辨率图像上的计算与内存开销呈二次增长,严重限制其应用。局部注意力通过限制关注邻域降低复杂度,但传统局部模式因窗口内令牌在1D序列中不连续,难以实现显著加速。本文提出一种基于希尔伯特曲线的新方法:先沿希尔伯特曲线重排图像令牌,再在重排后的1D序列上构建窗口与邻域。从块稀疏视角看,该策略显著提高块稀疏性,可与现有块稀疏核结合,提升2D局部注意力效率。实验表明,所提希尔伯特窗口注意力与希尔伯特滑动注意力分别实现约4倍和18倍加速。为验证实用性,构建了希尔伯特窗口Transformer与希尔伯特邻域Transformer,均实现端到端加速且精度损失极小。整体而言,结合希尔伯特引导局部注意力与块稀疏核,为图像2D局部注意力提供通用高效方案。

原文摘要 · Abstract (English)

The quadratic compute and memory costs of global self-attention severely limit its use in high-resolution images. Local attention reduces complexity by restricting attention to neighborhoods. Block-sparse kernels can further improve the efficiency of local attention, but conventional local attention patterns often fail to deliver significant speedups because tokens within a window are not contiguous in the 1D sequence. This work proposes a novel method for constructing windows and neighborhoods based on the Hilbert curve. Image tokens are first reordered along a Hilbert curve, and windows and neighborhoods are then formed on the reordered 1D sequence. From a block-sparse perspective, this strategy significantly increases block sparsity and can be combined with existing block-sparse kernels to improve the efficiency of 2D local attention. Experiments show that the proposed Hilbert Window Attention and Hilbert Slide Attention can accelerate window attention and slide attention by about $4\times$ and $18\times$, respectively. To assess practicality, the strategy is instantiated as the Hilbert Window Transformer and the Hilbert Neighborhood Transformer, both of which achieve end-to-end speedups with minimal accuracy loss. Overall, combining Hilbert-guided local attention with block-sparse kernels offers a general and practical approach to enhancing the efficiency of 2D local attention for images.

注意力机制图像处理稀疏计算加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。