通过在线剪枝减少3D问答中的冗余视觉令牌，提升效率与性能。
Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answering
- 利用深度和相机位姿将帧投影到共享体素空间，识别重叠区域并剪枝冗余图像令牌。
- 在不额外训练的前提下，最多降低50%的令牌使用量，且在多个基准测试中表现更优。
- 适合需要高效处理3D多视角输入的视觉语言模型应用，如智能机器人与数字孪生。
近期多模态大语言模型在2D问答任务中表现卓越，但扩展至3D问答仍具挑战，因通常需多视角输入，导致推理时计算开销大。现有方法依赖预先处理的帧选择或令牌合并算法，属离线方式。本文提出首个可无缝集成至现有多模态大模型的在线令牌剪枝方法，无需额外训练且内存占用更低。核心思想是利用深度信息与相机位姿将各输入帧投影至共享体素空间，识别跨帧的空间重叠区域，并在进入语言模型前剔除冗余图像令牌。该方法实现高效在线处理，最多减少50%令牌使用量。我们在Qwen2.5-VL-7B与Qwen3-VL-8B上验证，其在ScanQA、SQA3D和OpenEQA-HM3D基准上均取得更好性能。
原文摘要 · Abstract (English)
Recent Multi-modal Large Language Models (MLLMs) have demonstrated remarkable performance on 2D question answering tasks. However, extending these models to the 3D question answering remains challenging, as they typically require multiple views of the scene, which incurs substantial computational cost at inference. To mitigate this issue, existing solutions rely on strategic frame selection or token-merging algorithms that require preprocessing in advance all frames of the scene, i.e., an offline fashion. In contrast, we propose the first online token-pruning method that can be integrated seamlessly with current MLLM models for 3D question answering tasks, without additional training and with lower memory usage.Our key insight is to project each input frame into a shared voxel space using depth information and camera pose, identifying spatially-overlapped regions across frames and selectively pruning redundant image tokens before they enter the language model. Our method enables efficient online processing while reducing up to 50% of token usage. We apply this approach to Qwen2.5-VL-7B and Qwen3-VL-8B, demonstrating improved performance on the ScanQA, SQA3D, and OpenEQA-HM3D benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。