让冻结的视觉语言模型通过经验积累自我进化,无需外部工具
Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence

- 用验证反馈提炼空间经验为可复用知识
- 在5个基准上超越多数方法,4个基线模型全胜
- 适合追求零参数更新的智能体部署场景
空间智能正成为具身智能体、机器人规划和多模态助手的基础。现有方法主要分为两类:一类是微调或强化学习等后训练方法;另一类采用智能体范式,调用深度估计、3D重建等外部空间工具获取中间证据。本文探索一种互补且未被充分研究的路径:能否让冻结的视觉语言模型(VLM)在不依赖外部专家空间工具的前提下,通过参数无更新的自我演化提升空间推理能力?我们提出空间记忆智能体(SMA),一个基于经验的运行时框架,将经过验证的空间经验转化为可复用的迁移性知识。在可验证空间环境中,SMA 查询冻结的 VLM,获取预测答案与奖励,并通过验证器引导的反思机制,从空间经验中提炼出紧凑的可迁移知识。SMA 为每条知识分配一个转移可靠性评分(TRS),初始均匀设定,后续根据未来检索结果中的访问证据进行校准。在只读部署阶段,SMA 通过语义过滤与相似度-TRS联合排序检索知识,使记忆引导冻结模型推理。在五个代表性空间基准和四个基础 VLM 上,SMA 在每个基线模型块中均取得最高宏平均分,在20次评估中多数情况下达到最佳准确率,证明了其在不同规模冻结模型与环境下的实用化参数无更新空间自进化路径。
原文摘要 · Abstract (English)
Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLM agents, existing work has mainly followed two lines. One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools, such as depth estimation and 3D reconstruction tools, to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbf{parameter-update-free self-evolution}, without depending on external expert spatial tools at inference time? We present \textbf{Spatial Memory Agent (SMA)}, an \textbf{experience-grounded runtime framework} that converts verified spatial experience into reusable transferable lessons. In a verifiable spatial environment, SMA queries the frozen VLM, obtains a predicted answer and reward, and uses \textbf{verifier-guided reflection} to distill compact transferable lessons from spatial experience. SMA further assigns each lesson a \textbf{Transfer Reliability Score (TRS)}, which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During \textbf{read-only deployment}, SMA retrieves lessons by semantic filter and similarity-TRS combined ranking, allowing the retrieved memory to guide frozen model inference. Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy among the evaluated methods in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。