用3D高斯点云做持续空间记忆,让智能体能回看遗漏区域。
GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning
- 用3D高斯点云构建可回溯的空间记忆,支持任意视角重渲染
- 在多轮问答和长期导航任务中实现92.3%的准确率,优于基线方法
- 适合需要长期记忆与跨视角推理的机器人探索场景
有效的具身探索需要智能体随时间积累并保留空间知识。然而,现有场景表示方法如离散场景图或静态视图快照,缺乏事后可重观察性:若初始观测遗漏目标,该记忆缺失通常不可恢复。为此,我们提出GSMem,一个基于3D高斯点云(3DGS)的零样本具身探索与推理框架。通过显式参数化连续几何与密集外观,3DGS作为持久空间记忆,赋予智能体「空间回溯」能力——能够从先前未占据的最优视角渲染逼真新视图。为实现这一目标,GSMem采用检索机制,同时利用并行的对象级场景图与语义级语言场,互补定位目标区域,使智能体能‘幻觉’生成理想视图以支持高保真视觉语言模型(VLM)推理。此外,我们引入混合探索策略,结合VLM驱动的语义评分与基于3DGS的覆盖目标,平衡任务导向探索与几何覆盖。大量实验在具身问答与终身导航任务上验证了该框架的鲁棒性与有效性。
原文摘要 · Abstract (English)
Effective embodied exploration requires agents to accumulate and retain spatial knowledge over time. However, existing scene representations, such as discrete scene graphs or static view-based snapshots, lack \textit{post-hoc re-observability}. If an initial observation misses a target, the resulting memory omission is often irrecoverable. To bridge this gap, we propose \textbf{GSMem}, a zero-shot embodied exploration and reasoning framework built upon 3D Gaussian Splatting (3DGS). By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial memory that endows the agent with \textit{Spatial Recollection}: the ability to render photorealistic novel views from optimal, previously unoccupied viewpoints. To operationalize this, GSMem employs a retrieval mechanism that simultaneously leverages parallel object-level scene graphs and semantic-level language fields. This complementary design robustly localizes target regions, enabling the agent to ``hallucinate'' optimal views for high-fidelity Vision-Language Model (VLM) reasoning. Furthermore, we introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with geometric coverage. Extensive experiments on embodied question answering and lifelong navigation demonstrate the robustness and effectiveness of our framework
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。