arXiv:2606.19253cs.CVcs.AI2026-06

用全景投影统一多视角特征,让模型轻松理解3D空间关系。

OneCanvas: 3D Scene Understanding via Panoramic Reprojection

  • 将多视角图像块投影到统一全景坐标系,保留真实空间位置。
  • 在SQA3D和VSI-Bench上达到当前最佳性能,训练开销低一个数量级。
  • 适合机器人、具身智能等需要从特定视角推理的场景。

现有视觉语言模型在3D场景理解中依赖复杂几何编码器或大量训练预算。OneCanvas将所有视角的图像块通过深度与相机位姿反投影至3D世界坐标,直接放置于等距圆柱全景图上,按连续经度纬度定位,不进行重采样或融合。为恢复丢失的深度信息,在特征中加入基于度量坐标的3D位置嵌入。所有帧共享同一空间坐标系,无需修改主干网络。该表示可被预训练视觉语言模型直接处理。由于全景图可任意中心化,支持从特定视角进行情境化推理,满足机器人与具身智能需求。我们进一步设计空间预训练课程:在空白画布上程序化放置来自真实图像的对象特征块,生成覆盖广泛空间推理任务的动态监督信号,答案分布可控以减少空间捷径。OneCanvas在SQA3D和VSI-Bench上表现最优,对SPBench分布外数据有良好泛化能力,训练计算量仅为最强对比方法的十分之一。

原文摘要 · Abstract (English)

Existing approaches to 3D scene understanding in Vision-Language Models (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial reasoning. Instead, OneCanvas aggregates patch features from all views onto a single equirectangular panoramic canvas. Namely, each patch is unprojected to a 3D world coordinate using its depth and camera pose, then placed on the canvas at the continuous longitude and latitude of that point as seen from the canvas origin, with no rasterization or aggregation across overlapping views. A 3D position embedding of the patch's metric coordinates is added to its feature, restoring the depth lost when collapsing the world position to an angular canvas coordinate. Patches from all frames thus share one spatial coordinate system with no fusion or major architectural modifications of the backbone. The pretrained VLM consumes this representation as if it were an ordinary image. Because the canvas can be centered on any pose of interest, the same representation directly supports situated reasoning from a specific viewpoint, a common requirement in robotics and embodied AI. Thanks to this representation, we can also introduce a spatial pretraining curriculum: by procedurally placing patch features of objects, drawn from real images, at chosen 3D world positions on an otherwise empty canvas, we generate on-the-fly supervision spanning a broad range of spatial reasoning tasks, with answer distributions controlled to reduce spatial reasoning shortcuts. OneCanvas achieves state-of-the-art accuracy on SQA3D and VSI-Bench, and generalizes to out-of-distribution data on SPBench, using an order of magnitude less training compute than the strongest competing methods.

3D理解视觉语言模型全景投影具身智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。