arXiv:2509.21764cs.CVcs.LG2025-09

提出空间保持的令牌合并方法,适配多种视觉模型并提速不降精度。

CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbones

  • 采用二维结构化缩减与位置感知合并策略,保留空间布局。
  • 在SAM-H上提速1.25倍,仅损失0.7% mIOU;DeiT-B提速1.15倍,精度零下降。
  • 适用于窗口注意力、相对位置编码等空间结构模型,适合部署加速场景。

许多现代视觉变换器(ViT)骨干网络采用空间架构设计,如窗口注意力、SAM中的分解相对位置嵌入以及DINOv3中的旋转位置编码(RoPE)。这些架构对令牌压缩提出了新挑战,因现有多数方法无法保持其依赖的空间结构。本文提出一种简单而有效的令牌合并方法,能保持空间完整性,实现与空间架构的无缝兼容。该方法协调了两个看似矛盾的需求:(i) 利用空间布局中信息分布不均的特性,(ii) 合并后仍保持空间结构。我们的方法包含:(i) 二维缩减策略以强制结构化令牌布局,(ii) 空间感知合并算法维持相对令牌位置,(iii) 一种新的按维度最大幅值令牌表示,以保留显著特征。该方法在离线使用和微调下均表现优异,在多种视觉任务中达到空间与非空间架构的最先进性能。具体而言,在SAM-H上实现1.25倍加速,仅导致COCO上0.7%的mIOU下降;在DeiT-B上实现1.15倍加速,经过单轮微调后,ImageNet上准确率无下降。

原文摘要 · Abstract (English)

Many modern ViT backbones adopt spatial architectural designs, such as window attention, decomposed relative positional embeddings in SAM, and RoPE in DINOv3. Such architectures impose new challenges on token reduction, as the vast majority of existing methods fail to preserve the spatial structure these architectures depend on. In this paper, we introduce a simple yet effective token merging method that maintains spatial integrity, enabling seamless compatibility with spatial architectures. We reconcile two seemingly conflicting requirements: (i)exploiting the uneven information distribution across the spatial layout while (ii)preserving the spatial structure post-merging. Our approach employs (i)a 2D reduction strategy to enforce structured token layouts, (ii)a spatial-aware merging algorithm that maintains relative token positions, and (iii)a novel max-magnitude-per-dimension token representation that preserves salient features. Our method demonstrates strong performance both off-the-shelf and with fine-tuning, achieving state-of-the-art results on spatial and non-spatial architectures across various vision tasks. Specifically, we achieve 1.25x speedup on SAM-H with only 0.7% mIOU drop evaluated on COCO off-the-shelf, and 1.15x speedup on DeiT-B with no top-1 accuracy drop on ImageNet within just one epoch of fine-tuning.

视觉模型令牌合并ViT加速空间结构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。