arXiv:2603.29437cs.CV2026-03

通过语义与几何双重引导,大幅减少3D问答的视觉令牌数量。

SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering

  • 基于注意力机制筛选关键语义令牌,保留重要物体信息。
  • 结合3D几何距离补充空间多样性,确保场景覆盖完整。
  • 在仅10%视觉令牌下仍保持高推理精度,适合资源受限场景。

视觉语言模型(VLM)广泛应用于3D问答任务。典型流程中,多视角图像提取的视觉令牌与语言令牌拼接后由大语言模型联合处理。然而,多视角融合带来严重令牌冗余,导致视觉令牌集过大,限制了在有限令牌预算下的推理效率。视觉令牌剪枝成为主流解决方案。但现有方法多针对2D输入或依赖间接几何线索,难以显式保留语义关键物体并维持足够的空间覆盖以支持稳健的3D推理。本文提出SeGPruner,一种面向多视角图像的语义感知与几何引导的视觉令牌压缩框架。首先,通过基于注意力的重要性模块(语义感知令牌选择器)保留语义显著令牌,确保关键物体证据不丢失;其次,通过几何引导的选择器(几何感知令牌多样化器)补充空间上多样化的令牌,综合考虑语义相关性与3D几何距离。该机制协同实现对象级证据保留与全局场景覆盖,在激进令牌缩减下保持平衡。在ScanQA和OpenEQA数据集上的大量实验表明,SeGPruner显著提升推理效率:视觉令牌预算降低91%,推理延迟减少86%,同时在3D推理任务中保持竞争力。

原文摘要 · Abstract (English)

Vision-language models (VLMs) have been widely adopted for 3D question answering (3D QA). In typical pipelines, visual tokens extracted from multiple viewpoints are concatenated with language tokens and jointly processed by a large language model (LLM) for inference. However, aggregating multi-view observations inevitably introduces severe token redundancy, leading to an overly large visual token set that significantly hinders inference efficiency under constrained token budgets. Visual token pruning has emerged as a prevalent strategy to address this issue. Nevertheless, most existing pruners are primarily tailored to 2D inputs or rely on indirect geometric cues, which limits their ability to explicitly retain semantically critical objects and maintain sufficient spatial coverage for robust 3D reasoning. In this paper, we propose SeGPruner, a semantic-aware and geometry-guided token reduction framework for efficient 3D QA with multi-view images. Specifically, SeGPruner first preserves semantically salient tokens through an attention-based importance module (Saliency-aware Token Selector), ensuring that object-critical evidence is retained. It then complements these tokens with spatially diverse ones via a geometry-guided selector (Geometry-aware Token Diversifier), which jointly considers semantic relevance and 3D geometric distance. This cooperation between saliency preservation and geometry-guided diversification balances object-level evidence and global scene coverage under aggressive token reduction. Extensive experiments on ScanQA and OpenEQA demonstrate that SeGPruner substantially improves inference efficiency, reducing the visual token budget by 91% and inference latency by 86%, while maintaining competitive performance in 3D reasoning tasks.

3D问答令牌剪枝视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。