让机器人在操作中实时学习物理知识，不改模型也能变聪明。
PhysMem: Scaling Test-Time Memory for Embodied Physical Reasoning
- 测试时通过交互记录经验，生成并验证假设
- 真实任务中成功率从23%提升至76%
- 适合需要灵活应对物理变化的机器人场景
可靠的对象操作需要理解物体与环境间的物理属性差异。视觉语言模型（VLM）规划器虽能泛化地推理摩擦与稳定，但难以预测特定球体在特定表面的滚动行为，或判断某块石头能否作为稳定基底，除非有直接经验。我们提出PhysMem，一种测试时记忆框架，使VLM机器人规划器能在不更新模型参数的前提下，从实时交互中学习物理原理。系统记录交互经验，生成候选假设，并通过有针对性的交互验证，将确认的知识用于指导后续决策。核心设计是‘验证后再应用’：系统先用新观察检验假设，而非直接套用过往经验，从而降低对旧经验的僵化依赖，适应物理条件变化。我们在三个真实世界操作任务及四个VLM骨干模型的仿真基准上评估了PhysMem。在控制良好的砖块插入任务中，基于原理的抽象方法实现76%成功率，而直接经验检索仅为23%；真实实验显示，在长达30分钟的部署中持续表现更优。
原文摘要 · Abstract (English)
Reliable object manipulation requires understanding physical properties that vary across objects and environments. Vision-language model (VLM) planners can reason about friction and stability in general terms; however, they often cannot predict how a specific ball will roll on a particular surface or which stone will provide a stable foundation without direct experience. We present PhysMem, a memory framework that enables VLM robot planners to learn physical principles from interaction at test time, without updating model parameters. The system records experiences, generates candidate hypotheses, and verifies them through targeted interaction before promoting validated knowledge to guide future decisions. A central design choice is verification before application: the system tests hypotheses against new observations rather than applying retrieved experience directly, reducing rigid reliance on prior experience when physical conditions change. We evaluate PhysMem on three real-world manipulation tasks and simulation benchmarks across four VLM backbones. On a controlled brick insertion task, principled abstraction achieves 76% success compared to 23% for direct experience retrieval, and real-world experiments show consistent improvement over 30-minute deployment sessions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。