用视觉词压缩图像块，实时推理更省电。
Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformers
- 按图像内/间相似性合并高频视觉子块为视觉词
- 能耗降低最高达47%，优于8位量化和令牌合并
- 无需训练，适合实时推理场景
视觉变换器的部署成本正成为工业应用的主要障碍。现有压缩技术需额外端到端微调或显著降低能效，不适用于在线推理（实时处理新输入）。我们提出无需训练的视觉词分词器（Visual-Word Tokenizer, VWT），通过利用图像内或图像间统计信息，将频繁出现的视觉子块（图像补丁）合并为视觉词，低频部分保持不变。实验表明，能耗最高可降低47%；相比之下，8位量化与令牌合并方法可能使能耗增加高达500%以上。结果表明，VWT在性能略有下降的前提下，非常适合高效在线推理。论文代码已公开。
原文摘要 · Abstract (English)
The cost of deploying vision transformers increasingly represents a barrier to wider industrial adoption. Existing compression techniques require additional end-to-end fine-tuning or incur a significant drawback to energy efficiency, making them ill-suited for online (real-time) inference, where a prediction is made on any new input as it comes in. We introduce the $\textbf{Visual-Word Tokenizer}$ (VWT), a training-free method for reducing energy costs while retaining performance. The VWT groups visual subwords (image patches) that are frequently used into visual words, while infrequent ones remain intact. To do so, $\textit{intra}$-image or $\textit{inter}$-image statistics are leveraged to identify similar visual concepts for sequence compression. Experimentally, we demonstrate a reduction in energy consumed of up to 47%. Comparative approaches of 8-bit quantization and token merging can lead to significantly increased energy costs (up to 500% or more). Our results indicate that VWTs are well-suited for efficient online inference with a marginal compromise on performance. The experimental code for our paper is also made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。