用正交性筛选视觉令牌,高效降低高分辨率视觉语言模型推理开销。
GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models

- 基于令牌间正交性增量选择,不依赖训练或查询信息。
- 在五个模型、十一个基准上平均性能保留率优于最强基线。
- 适合追求低延迟推理的高分辨率视觉语言模型部署场景。
现代视觉语言模型(VLMs)越来越多地采用动态或高分辨率视觉编码,生成数千个视觉令牌,显著增加下游语言模型的推理成本。现有令牌压缩方法通过令牌重要性、查询相关性、覆盖度、成对多样性或子集级目标评估令牌价值。本文核心洞察是:从已选令牌子集的互补性角度看待视觉令牌压缩——不孤立评分或依赖成对关系,而是评估新令牌特征与当前保留子集跨度之间的正交能量。基于此,提出无需训练、与查询无关的贪心正交令牌选择(GOTS)方法。每一步选择与当前保留子集正交性最大的令牌,该规则精确最大化候选添加项的增广格拉姆行列式,为每一步扩展提供明确的局部几何保证。在来自Qwen-VL和InternVL系列的五个高分辨率VLM骨干网络及十一个多样化基准上,GOTS的平均性能保留率高于最强基线;受控的OCRBench实验表明,在扣除选择开销后,其降低了模型侧的首令牌延迟。代码已开源:https://github.com/newLLing/GOTS。
原文摘要 · Abstract (English)
Modern vision-language models (VLMs) increasingly rely on dynamic or high-resolution visual encoding, producing thousands of visual tokens that substantially increase downstream language-model inference cost. Existing token-reduction methods assess token utility through token-wise importance, query relevance, coverage, pairwise diversity, or subset-level objectives. Our key insight is to view visual token reduction through selected-span complementarity: instead of scoring a token in isolation or through pairwise relations, we assess how much of its feature is orthogonal to the span of the already retained subset. Based on this perspective, we propose Greedy Orthogonal Token Selection (GOTS), a training-free and query-agnostic method. At each step, GOTS selects the token with the largest residual energy orthogonal to the current retained span. This rule exactly maximizes the one-step augmented Gram determinant among candidate additions, giving each greedy step a precise local geometric guarantee for subset expansion. Across five high-resolution VLM backbones from the Qwen-VL and InternVL families and eleven diverse benchmarks, GOTS achieves higher average performance retention than the strongest evaluated baselines, and a controlled OCRBench study shows that it reduces model-side time-to-first-token after accounting for selection overhead. Code is available at https://github.com/newLLing/GOTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。