arXiv:2602.03060cs.CV2026-02中稿 · ICLR被引 7

发现视觉大模型隐含坐标系,实现不降性能的高效图像标记剪枝。

IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning

  • 通过分析旋转位置编码,找到对空间推理至关重要的隐式坐标标记。
  • 剪掉约一半视觉标记,性能保持99%以上,部分任务还提升。
  • 无需训练、适配多种视觉语言模型,适合高分辨率图像处理场景。

大型视觉语言模型(LVLMs)在多任务上表现优异,但处理高分辨率图像时推理成本过高。视觉标记剪枝虽有潜力,但现有方法多关注语义相关性,常误删影响空间推理的关键标记。本文揭示了LVLM通过旋转位置编码(RoPE)隐式构建视觉坐标系统,其中特定位置的标记为关键的隐式视觉坐标(IVC)标记。基于此,提出无需训练、响应提示的剪枝策略IVC-Prune,保留IVC标记与语义相关的前景标记。IVC标记通过分析RoPE矩阵性质确定,对应旋转矩阵趋近于单位阵或90°旋转的情况;前景标记则通过两阶段流程识别:先找语义种子,再用值向量相似度细化上下文。在四类代表性LVLM和二十个基准上的评估显示,IVC-Prune可将视觉标记减少约50%,性能保持≥99%,并在多个任务上实现提升。代码已开源。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) achieve impressive performance across multiple tasks. A significant challenge, however, is their prohibitive inference cost when processing high-resolution visual inputs. While visual token pruning has emerged as a promising solution, existing methods that primarily focus on semantic relevance often discard tokens that are crucial for spatial reasoning. We address this gap through a novel insight into \emph{how LVLMs process spatial reasoning}. Specifically, we reveal that LVLMs implicitly establish visual coordinate systems through Rotary Position Embeddings (RoPE), where specific token positions serve as \textbf{implicit visual coordinates} (IVC tokens) that are essential for spatial reasoning. Based on this insight, we propose \textbf{IVC-Prune}, a training-free, prompt-aware pruning strategy that retains both IVC tokens and semantically relevant foreground tokens. IVC tokens are identified by theoretically analyzing the mathematical properties of RoPE, targeting positions at which its rotation matrices approximate identity matrix or the $90^\circ$ rotation matrix. Foreground tokens are identified through a robust two-stage process: semantic seed discovery followed by contextual refinement via value-vector similarity. Extensive evaluations across four representative LVLMs and twenty diverse benchmarks show that IVC-Prune reduces visual tokens by approximately 50\% while maintaining $\geq$ 99\% of the original performance and even achieving improvements on several benchmarks. Source codes are available at https://github.com/FireRedTeam/IVC-Prune.

视觉剪枝空间推理大模型优化视觉坐标

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。