用深度图辅助视觉令牌合并，提升ViT加速效果
ToSA: Token Merging with Spatial Awareness
- 引入深度图生成空间伪令牌，融合空间信息指导合并
- 在多个视觉与具身问答任务上优于现有方法，运行时间大幅降低
- 适合需要高效推理的视觉模型部署场景
令牌合并已成为加速视觉变换器（ViT）的有效策略，通过减少计算开销实现。然而，现有方法主要依赖视觉令牌的特征相似性进行合并，忽略了早期层中弱视觉信息下空间信息的潜在价值。本文提出ToSA，一种结合语义与空间感知的新型令牌合并方法。ToSA利用深度图像生成伪空间令牌，作为视觉令牌合并过程中的辅助空间信息。引入空间感知后，合并策略更具信息量，能更好保留关键场景结构。实验表明，ToSA在多个视觉与具身问答基准测试中均优于先前方法，同时显著降低ViT运行时间，为ViT加速提供高效解决方案。代码将发布于：https://github.com/hsiangwei0903/ToSA
原文摘要 · Abstract (English)
Token merging has emerged as an effective strategy to accelerate Vision Transformers (ViT) by reducing computational costs. However, existing methods primarily rely on the visual token's feature similarity for token merging, overlooking the potential of integrating spatial information, which can serve as a reliable criterion for token merging in the early layers of ViT, where the visual tokens only possess weak visual information. In this paper, we propose ToSA, a novel token merging method that combines both semantic and spatial awareness to guide the token merging process. ToSA leverages the depth image as input to generate pseudo spatial tokens, which serve as auxiliary spatial information for the visual token merging process. With the introduced spatial awareness, ToSA achieves a more informed merging strategy that better preserves critical scene structure. Experimental results demonstrate that ToSA outperforms previous token merging methods across multiple benchmarks on visual and embodied question answering while largely reducing the runtime of the ViT, making it an efficient solution for ViT acceleration. The code will be available at: https://github.com/hsiangwei0903/ToSA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。