用视觉语言模型实现无需训练的单目场景配准，解决机器人和AR的时空记忆难题。
PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models
- 利用预训练视觉语言模型构建物体级3D场景图，实现无深度传感器的场景理解。
- 通过物体高度先验与成对相似性验证，提升跨视角物体匹配准确率。
- 零参数设计，直接迁移至下游任务，适用于真实手持拍摄的模糊动态视频。
在不同时间拍摄的同一室内空间之间进行配准，是机器人和增强现实系统实现持久空间记忆的基础。然而，现实场景中该任务具有第一人称视角且仅依赖RGB图像，头戴摄像头产生的画面模糊、运动快速、视域重叠有限，难以恢复稠密几何结构。传统方法依赖高质量点云，而现有学习型场景图方法需预先构建或标注图谱及训练匹配器，在第一人称数据下表现脆弱。本文提出PROSE（Prompted Scene rEgistration），使用预训练视觉语言模型作为场景理解与跨扫描匹配的统一来源。将每个RGB序列通过现成基础模型（几何、分割、语言）转化为物体级3D场景图，再以提示词驱动同一VLM匹配两序列中的物体实例。为确保匹配可靠性，引入物体高度作为先验，并采用成对‘相同/不同’查询验证每组匹配；最终通过假设每个匹配对象的候选刚性变换，选择几何一致性最强的解。PROSE不引入任何可学习参数，无需深度传感器、训练或标注图谱。在第一人称数据集Aria Digital Twin与Aria Everyday Activities上，其在真实点云和仅由RGB重建的点云上均优于几何与学习型基线，生成的场景图可直接用于下游任务。
原文摘要 · Abstract (English)
Registering two captures of the same indoor space taken at different times underpins persistent spatial memory for robots and AR systems, yet the realistic version of this task is egocentric and its most scalable form is RGB-only. Head-mounted cameras yield blurry, fast-moving, partially overlapping views from which dense geometry is hard to recover. Classical registration leans on exactly the clean point clouds this setting lacks, while learned scene-graph methods require a pre-built or annotated graph and a trained matcher that we find brittle under egocentric data. We take a different route, using a pretrained vision-language model as the source of both scene understanding and cross-scan matching. Our method, PROSE (Prompted Scene rEgistration), lifts each RGB sequence into an object-level 3D scene graph using off-the-shelf foundation models for geometry, segmentation, and language, then prompts the same VLM to match object instances across the two RGB sequences. To make this matching tractable and reliable, we leverage object heights as a prior and verify each proposed match with a paired same/different query, then solve for the rigid transform by hypothesizing a candidate per matched object and selecting the one with the strongest geometric consensus. PROSE adds no learned parameters and requires no depth sensor, training, or annotated graph. On the egocentric Aria Digital Twin and Aria Everyday Activities benchmarks, it outperforms both geometric and learned scene-graph baselines in registration accuracy, on ground-truth and RGB-reconstructed point clouds alike, and the scene graph it produces transfers directly to downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。