动态选精简令牌，提升视觉Transformer效率与精度
Context-Aware Token Selection and Packing for Enhanced Vision Transformer
- 用轻量门控层动态挑选关键令牌，替代全量计算
- 目标检测提升0.6 mAP，计算成本降低16.4%
- 适合追求高效推理的视觉任务应用
近年来，视觉变换器的长程注意力机制在多种计算机视觉任务中推动了显著性能突破。然而，传统自注意力机制会处理所有令牌，包括无信息量的冗余内容，导致效率低下且结果不准确。尽管已有稀疏注意力机制通过裁剪令牌缓解该问题，但其缺乏上下文感知能力，常采用统一策略处理不同输入，或仅优化推理阶段效率。为此，我们提出一种新算法：选并打包注意力（SPA）。SPA利用低成本门控层动态选择信息丰富令牌，并将其打包成新批次，支持在并行化GPU训练与推理中使用可变数量的令牌。在多个数据集和视觉任务上的广泛实验表明，SPA实现更优性能与效率，目标检测提升0.6 mAP，计算成本降低16.4%。
原文摘要 · Abstract (English)
In recent years, the long-range attention mechanism of vision transformers has driven significant performance breakthroughs across various computer vision tasks. However, the traditional self-attention mechanism, which processes both informative and non-informative tokens, suffers from inefficiency and inaccuracies. While sparse attention mechanisms have been introduced to mitigate these issues by pruning tokens involved in attention, they often lack context-awareness and intelligence. These mechanisms frequently apply a uniform token selection strategy across different inputs for batch training or optimize efficiency only for the inference stage. To overcome these challenges, we propose a novel algorithm: Select and Pack Attention (SPA). SPA dynamically selects informative tokens using a low-cost gating layer supervised by selection labels and packs these tokens into new batches, enabling a variable number of tokens to be used in parallelized GPU batch training and inference. Extensive experiments across diverse datasets and computer vision tasks demonstrate that SPA delivers superior performance and efficiency, including a 0.6 mAP improvement in object detection and a 16.4% reduction in computational costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。