用语义聚类生成高效3D代理表示，提升视觉语言模型空间推理能力。
Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment
- 通过语义与几何编码器提取特征，进行语义感知聚类生成3D代理
- 在短视频序列下，3D视觉问答等任务达到领先或顶尖性能
- 适合需要高效3D理解的视觉语言模型研究者使用
视觉语言模型（VLM）的空间智能因在三维世界中推理的实际需求而受到关注。尽管已有进展，多数方法仍沿用传统2D流程，采用像素对齐的视觉表示。然而，依赖对应关系的模型常缺乏空间一致性，而带有3D几何先验的表示模型在视觉序列化时效率较低。为此，我们提出Proxy3D方法，构建紧凑且全面的3D代理表示。仅需视频帧输入，通过语义与几何编码器提取场景特征，并进行语义感知聚类，在3D空间中获得一组代理。为实现表示对齐，我们构建了SpaceSpan数据集，并采用多阶段训练，将所提3D代理表示融入VLM。在使用较短视觉序列的情况下，该方法在3D视觉问答、视觉定位及通用空间智能基准测试中表现优异，达到竞争性或最先进水平。
原文摘要 · Abstract (English)
Spatial intelligence in vision-language models (VLMs) attracts research interest with the practical demand to reason in the 3D world.Despite promising results, most existing methods follow the conventional 2D pipeline in VLMs and use pixel-aligned representations for the vision modality. However, correspondence-based models with implicit 3D scene understanding often fail to achieve spatial consistency, and representation-based models with 3D geometric priors lack efficiency in vision sequence serialization. To address this, we propose a Proxy3D method with compact yet comprehensive 3D proxy representations for the vision modality. Given only video frames as input, we employ semantic and geometric encoders to extract scene features and then perform their semantic-aware clustering to obtain a set of proxies in the 3D space. For representation alignment, we further curate the SpaceSpan dataset and apply multi-stage training to adopt the proposed 3D proxy representations with the VLM. When using shorter sequences for vision information, our method achieves competitive or state-of-the-art performance in 3D visual question answering, visual grounding and general spatial intelligence benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。