让机器人在动态环境中实时感知并定位物品，成功率超70%。
DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation
- 用动态时空语义记忆构建3D环境表示，支持持续更新。
- 对非静态物体的抓取成功率达70%，性能提升超2倍。
- 适合真实世界中环境变化频繁的移动操作任务。
开放词汇移动操作领域取得了显著进展，目标是使机器人能在任何环境中根据自然语言指令完成任务。然而，当前多数系统假设环境静态，限制了其在真实场景中的应用，因为环境常因人类活动或机器人自身操作而改变。本文提出DynaMem，一种用于开放世界移动操作的新方法，利用动态时空语义记忆来表征机器人环境。DynaMem构建三维数据结构，维护点云的动态记忆，并通过多模态大模型或前沿视觉-语言模型生成的开放词汇特征，回答开放词汇物体定位查询。借助DynaMem，机器人可在新环境中探索、搜索记忆中未记录的物体，并在物体移动、出现或消失时持续更新记忆。我们在Stretch SE3机器人上于三个真实场景和九个离线场景进行大量实验，对非静态物体的平均抓取-放置成功率达70%，较现有静态系统提升超过2倍。代码及实验与部署视频已开源，详见项目网站：https://dynamem.github.io/
原文摘要 · Abstract (English)
Significant progress has been made in open-vocabulary mobile manipulation, where the goal is for a robot to perform tasks in any environment given a natural language description. However, most current systems assume a static environment, which limits the system's applicability in real-world scenarios where environments frequently change due to human intervention or the robot's own actions. In this work, we present DynaMem, a new approach to open-world mobile manipulation that uses a dynamic spatio-semantic memory to represent a robot's environment. DynaMem constructs a 3D data structure to maintain a dynamic memory of point clouds, and answers open-vocabulary object localization queries using multimodal LLMs or open-vocabulary features generated by state-of-the-art vision-language models. Powered by DynaMem, our robots can explore novel environments, search for objects not found in memory, and continuously update the memory as objects move, appear, or disappear in the scene. We run extensive experiments on the Stretch SE3 robots in three real and nine offline scenes, and achieve an average pick-and-drop success rate of 70% on non-stationary objects, which is more than a 2x improvement over state-of-the-art static systems. Our code as well as our experiment and deployment videos are open sourced and can be found on our project website: https://dynamem.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。