用空间覆盖原理精简3D多视角视觉令牌,大幅减少冗余同时保持高推理性能。
CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs

- 基于令牌坐标设计确定性剪枝策略,不依赖训练或注意力权重。
- 仅用约8%的令牌即保留93.5%性能,优于现有最优方法3.9个百分点。
- 适用于多种视觉语言模型,可直接作为插件使用。
将3D场景表示为多视角图像,使2D视觉语言模型(VLMs)能复用预训练先验进行3D推理,避免3D标注数据稀缺问题。但该方式生成数千个冗余视觉令牌,成本随视角增加而上升。现有剪枝方法分为两类:基于重要性的学习方法根据注意力或编码特征排序,因冗余本质为空间性,导致保留多个相似区域的重复令牌,忽略大部分场景;体素化方法虽提升空间覆盖度,却无法精确控制令牌预算,且在多视角重叠时饱和,保留率远低于目标。本文发现空间覆盖度与3D推理性能正相关,提出CoVeR——一种无需训练、仅依赖令牌坐标的确定性选择器。CoVeR确保所选令牌在场景中均匀覆盖各区域,突破两类方法局限:实现每场景精确预算控制,打破体素化饱和瓶颈,避免近似重复选择。大量实验表明,CoVeR在三个3D推理基准上均超越当前最优,且作为即插即用模块在四个VLM上验证泛化能力。值得注意的是,仅使用约8%的视觉令牌,便保留93.5%的完整令牌性能,平均超越SOTA 3.9个百分点。
原文摘要 · Abstract (English)
Representing a 3D scene as multi-view images allows 2D VLMs to reason in 3D by reusing priors from pre-training, sidestepping the scarcity of annotated 3D data. However, it produces thousands of redundant visual tokens whose cost grows with every view. Existing visual token pruners fall into two families, each limited in the 3D multi-view setting. Learned importance methods rank tokens by attention or encoder features; because redundancy here is fundamentally spatial, they keep near-duplicate tokens from a few prominent regions and leave most of the scene unrepresented. Voxelization methods improve spatial coverage but cannot enforce an exact token budget and saturate as multi-view observations overlap in 3D, capping retention well below the target. We show that spatial coverage is associated with 3D reasoning performance and introduce CoVeR, a deterministic, training-free selector that uses only token coordinates, with no learned signals. CoVeR selects tokens that collectively cover every region of the scene, and solves the limitations of both families: it enforces an exact per-scene budget, breaks the voxelization saturation plateau, and avoids the near-duplicate selections of learned importance. Extensive experiments show CoVeR outperforms prior SOTAs on all three 3D reasoning benchmarks and generalizes as a plug-and-play module tested across four VLMs. Notably, with only $\approx$8% of visual tokens, it preserves 93.5% of full-token performance, surpassing SOTA by 3.9 percentage points on average across benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。