用3D高斯点云生成语言可理解的3D表示，实现仅需图像的零样本3D视觉问答
SplatTalk: 3D VQA with Gaussian Splatting
- 将3D高斯点云转化为适合大模型输入的3D token
- 在多个基准上超越仅用图像的2D模型和专用3D模型
- 适用于机器人、AR/VR等需理解3D环境的场景
语言引导的3D场景理解对机器人、增强现实/虚拟现实及人机交互具有重要意义，使模型能通过自然语言理解与互动3D环境。尽管2D视觉语言模型在2D VQA任务中取得显著进展，但3D领域的进展因3D数据复杂性和人工标注成本高而受限。本文提出SplatTalk，一种基于通用3D高斯点云（3DGS）框架的新方法，可生成直接输入预训练大语言模型的3D token，实现仅需姿态图像的零样本3D视觉问答。在多个基准上的实验表明，该方法在性能上超越了专门训练的3D模型以及仅依赖图像的前代2D-LMM模型（在相同设置下），且与使用额外3D输入的最先进3D LMMs相当。项目主页：https://splat-talk.github.io/
原文摘要 · Abstract (English)
Language-guided 3D scene understanding is important for advancing applications in robotics, AR/VR, and human-computer interaction, enabling models to comprehend and interact with 3D environments through natural language. While 2D vision-language models (VLMs) have achieved remarkable success in 2D VQA tasks, progress in the 3D domain has been significantly slower due to the complexity of 3D data and the high cost of manual annotations. In this work, we introduce SplatTalk, a novel method that uses a generalizable 3D Gaussian Splatting (3DGS) framework to produce 3D tokens suitable for direct input into a pretrained LLM, enabling effective zero-shot 3D visual question answering (3D VQA) for scenes with only posed images. During experiments on multiple benchmarks, our approach outperforms both 3D models trained specifically for the task and previous 2D-LMM-based models utilizing only images (our setting), while achieving competitive performance with state-of-the-art 3D LMMs that additionally utilize 3D inputs. Project website: https://splat-talk.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。