arXiv:2507.09702cs.CV2025-07综述被引 2

系统梳理视觉Transformer的令牌压缩技术,评估其在边缘设备上的效果。

Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI

  • 按剪枝、合并等策略分类,统一比较不同压缩方法。
  • 标准ViT上压缩有效,但紧凑版模型性能下降明显。
  • 适合研究边缘AI中轻量化Transformer优化的开发者参考。

令牌压缩技术近年成为加速视觉Transformer推理的重要手段。由于注意力计算随序列长度呈二次增长,此类方法通过移除低信息量令牌来提升推理吞吐量。尽管已有大量研究探索大型ViT中的精度-效率权衡,仍存在两大空白:一是缺乏基于核心策略(如剪枝、合并或混合)与部署场景(微调或即插即用)的系统性分类对比;二是现有基准多集中于标准ViT(如ViT-B、ViT-L),未验证这些方法在结构压缩型Transformer上的有效性,而后者正广泛部署于资源受限的边缘设备。为此,本文首次构建了令牌压缩方法的系统性分类体系,并在标准与紧凑型ViT架构上评估代表性技术。实验表明,虽然令牌压缩在通用ViT上表现良好,但在紧凑设计中常出现性能下滑。该发现为未来针对边缘AI与AI代理的轻量化变压器优化研究提供了实践指导。

原文摘要 · Abstract (English)

Token compression techniques have recently emerged as powerful tools for accelerating Vision Transformer (ViT) inference in computer vision. Due to the quadratic computational complexity with respect to the token sequence length, these methods aim to remove less informative tokens before the attention layers to improve inference throughput. While numerous studies have explored various accuracy-efficiency trade-offs on large-scale ViTs, two critical gaps remain. First, there is a lack of unified survey that systematically categorizes and compares token compression approaches based on their core strategies (e.g., pruning, merging, or hybrid) and deployment settings (e.g., fine-tuning vs. plug-in). Second, most benchmarks are limited to standard ViT models (e.g., ViT-B, ViT-L), leaving open the question of whether such methods remain effective when applied to structurally compressed transformers, which are increasingly deployed on resource-constrained edge devices. To address these gaps, we present the first systematic taxonomy and comparative study of token compression methods, and we evaluate representative techniques on both standard and compact ViT architectures. Our experiments reveal that while token compression methods are effective for general-purpose ViTs, they often underperform when directly applied to compact designs. These findings not only provide practical insights but also pave the way for future research on adapting token optimization techniques to compact transformer-based networks for edge AI and AI agent applications.

视觉Transformer边缘AI令牌压缩轻量化模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。