用2D视觉语言模型做3D问答，仅靠选图就能达到顶尖效果
3D Question Answering via only 2D Vision-Language Models
- 从3D点云采样关键且多样的2D视角，输入现有2D模型
- 在ScanQA和SQA上超越现有方法，无需微调2D模型
- 适合想低成本做3D理解的研究者或应用开发者
大型视觉语言模型（LVLM）已推动多个领域发展。本文探索如何利用其能力解决3D场景理解任务，以3D问答（3D-QA）为例。由于3D训练数据有限，我们不训练LVLM，而是零样本推理：从3D点云中采样2D视图，输入2D模型作答。当使用如LLAVA-OV的2D模型时，采样视图质量至关重要。为此提出cdViews，一种自动选择关键且多样视图的方法。其包含两个核心组件：viewSelector基于提供答案特定信息的潜力筛选关键视图，viewNMS通过空间重叠判断移除冗余视图以增强多样性。在广泛使用的ScanQA和SQA基准上评估，cdViews仅依赖2D模型且无微调，即实现3D-QA的最先进性能。结果表明，当前2D LVLM是应对3D任务最有效的替代方案（相比资源密集型3D LVLM）。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) have significantly advanced numerous fields. In this work, we explore how to harness their potential to address 3D scene understanding tasks, using 3D question answering (3D-QA) as a representative example. Due to the limited training data in 3D, we do not train LVLMs but infer in a zero-shot manner. Specifically, we sample 2D views from a 3D point cloud and feed them into 2D models to answer a given question. When the 2D model is chosen, e.g., LLAVA-OV, the quality of sampled views matters the most. We propose cdViews, a novel approach to automatically selecting critical and diverse Views for 3D-QA. cdViews consists of two key components: viewSelector prioritizing critical views based on their potential to provide answer-specific information, and viewNMS enhancing diversity by removing redundant views based on spatial overlap. We evaluate cdViews on the widely-used ScanQA and SQA benchmarks, demonstrating that it achieves state-of-the-art performance in 3D-QA while relying solely on 2D models without fine-tuning. These findings support our belief that 2D LVLMs are currently the most effective alternative (of the resource-intensive 3D LVLMs) for addressing 3D tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。