不训练即可动态压缩视觉令牌,提升多模态模型推理效率。
OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

- 用寄存器锚定的相对证据测试替代固定排名,自适应剪枝冗余与相关性。
- 在LLaVA-NeXT上将2880个视觉令牌压缩至约40个,准确率保留93%以上。
- 无需训练,适用于不同图像和查询,适合资源受限场景部署。
视觉语言模型依赖长视觉令牌序列进行理解,导致预填充阶段计算与内存开销大。现有剪枝方法采用固定排名策略,按重要性保留前K个令牌,但该策略易受注意力汇聚影响,且因图像冗余与查询依赖性,固定预算不可靠。本文提出OccamToken,一种无需训练的框架,以寄存器锚定的相对证据测试取代绝对排名。其核心思想是:寄存器令牌自动吸收低信息注意力模式,可作为稳定参考,判断视觉令牌是否提供超出参考的信息。基于此,通过寄存器注意力动态设定阈值,实现图像自适应冗余剪枝与查询自适应相关性剪枝。在LLaVA-NeXT、LLaVA-v1.5和Qwen3-VL上均显著提升精度-效率权衡。尤其在LLaVA-NeXT中,将2880个视觉令牌压缩至约40个,仍保持超过93%原始准确率,实现极端1.4%保留率下的稳定压缩。
原文摘要 · Abstract (English)
Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。