让机器人通过看图和说话自我纠错,提升任务成功率。
LERa: Replanning with Visual Feedback in Instruction Following
- 用视觉反馈+自然语言重规划,仅需图像和指令即可纠错。
- 在动态环境和模拟失败场景中,成功率最高提升67%。
- 适合需要自主应对错误的机器人系统,尤其适合真实部署。
大型语言模型在机器人任务规划中应用日益广泛,但其对文本输入的依赖限制了对现实世界变化和执行失败的适应能力。为解决此问题,我们提出LERa——Look, Explain, Replan,一种基于视觉语言模型的重规划方法,利用视觉反馈实现自适应调整。与现有方法不同,LERa仅需原始RGB图像、自然语言指令、初始任务计划和失败检测信号,无需额外的物体检测或预定义条件。重规划过程包含三步:(i) Look——生成场景描述并识别错误;(ii) Explain——提供修正指导;(iii) Replan——相应修改任务计划。LERa可适配多种机器人架构,能处理动态场景变化和执行失败。我们在新提出的ALFRED-ChaOS和VirtualHome-ChaOS数据集上评估,动态环境下性能较基线提升40%。在PyBullet模拟器中的桌面操作任务中,任务失败概率固定时,成功率达67%。真实机器人实验也验证了其有效性。结果表明,LERa是机器人任务执行中具备误差感知能力的鲁棒且灵活的解决方案。
原文摘要 · Abstract (English)
Large Language Models are increasingly used in robotics for task planning, but their reliance on textual inputs limits their adaptability to real-world changes and failures. To address these challenges, we propose LERa - Look, Explain, Replan - a Visual Language Model-based replanning approach that utilizes visual feedback. Unlike existing methods, LERa requires only a raw RGB image, a natural language instruction, an initial task plan, and failure detection - without additional information such as object detection or predefined conditions that may be unavailable in a given scenario. The replanning process consists of three steps: (i) Look - where LERa generates a scene description and identifies errors; (ii) Explain - where it provides corrective guidance; and (iii) Replan - where it modifies the plan accordingly. LERa is adaptable to various agent architectures and can handle errors from both dynamic scene changes and task execution failures. We evaluate LERa on the newly introduced ALFRED-ChaOS and VirtualHome-ChaOS datasets, achieving a 40% improvement over baselines in dynamic environments. In tabletop manipulation tasks with a predefined probability of task failure within the PyBullet simulator, LERa improves success rates by up to 67%. Further experiments, including real-world trials with a tabletop manipulator robot, confirm LERa's effectiveness in replanning. We demonstrate that LERa is a robust and adaptable solution for error-aware task execution in robotics. The project page is available at https://lera-robo.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。