用切片法让2D视觉语言模型理解3D体素数据，提升空间感知能力。
VoxRep: Enhancing 3D Spatial Understanding in 2D Vision-Language Models via Voxel Representation
- 将体素空间沿主轴切片，用2D VLM逐层处理切片。
- 实现对物体身份、颜色、位置的3D语义提取，无需复杂3D网络。
- 适合机器人、自动驾驶等需3D理解的场景，可复用现有2D模型。
理解3D环境对机器人与自动驾驶等智能系统至关重要。体素网格提供了结构化的3D空间表示，但从中提取高层次语义仍具挑战。本文提出一种新方法，利用视觉语言模型（VLM）从体素数据中提取“体素语义”——包括物体身份、颜色和位置。关键在于，不采用复杂的3D网络，而是沿主轴（如Z轴，类比CT切片）系统性地切割体素空间，将得到的2D切片格式化后顺序输入标准VLM的图像编码器。模型学习在切片间聚合信息，并将空间模式与语言组件提供的语义概念关联。该切片策略旨在直接利用预训练2D VLM的能力，高效实现从体素表示到3D语义理解的映射。
原文摘要 · Abstract (English)
Comprehending 3D environments is vital for intelligent systems in domains like robotics and autonomous navigation. Voxel grids offer a structured representation of 3D space, but extracting high-level semantic meaning remains challenging. This paper proposes a novel approach utilizing a Vision-Language Model (VLM) to extract "voxel semantics"-object identity, color, and location-from voxel data. Critically, instead of employing complex 3D networks, our method processes the voxel space by systematically slicing it along a primary axis (e.g., the Z-axis, analogous to CT scan slices). These 2D slices are then formatted and sequentially fed into the image encoder of a standard VLM. The model learns to aggregate information across slices and correlate spatial patterns with semantic concepts provided by the language component. This slice-based strategy aims to leverage the power of pre-trained 2D VLMs for efficient 3D semantic understanding directly from voxel representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。