让机器人通过视觉和经验不断学习新技能，失败后自动重规划，成功则存为可复用技能。
ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learning
- 结合视觉感知与技能记忆，失败时重规划，成功时存为可复用动作序列。
- 在仿真与真实机器人上均提升任务成功率，实现稳定跨场景迁移。
- 适合长期自主学习的机器人系统，尤其对少样本新任务适应性强。
基于强化学习（RL）或模仿学习（IL）的机器人通常对新任务适应缓慢，而近期的大语言模型（LLMs）和视觉-语言模型（VLMs）虽能从少量数据中实现知识丰富的规划。然而将它们用于运动规划面临两大挑战：(i) 符号化计划往往未与场景几何和物体物理对齐；(ii) 相同提示下的模型输出存在波动，影响执行可靠性。我们提出 ViReSkill 框架，将视觉引导的重规划与技能记忆结合，用于基于大模型的持续学习。当任务失败时，重规划器根据当前场景生成新的动作序列；成功后，该计划作为可复用技能被存储，并在后续遇到相同情境时直接调用，无需再次调用大模型。这一反馈机制实现了自主持续学习：每次尝试立即扩展技能库并稳定后续执行。我们在 LIBERO、RLBench 等模拟器及实际机器人上进行了评估，结果表明，无论在仿真还是真实环境中，其任务成功率始终优于传统基线，展现出强大的模拟到现实泛化能力。
原文摘要 · Abstract (English)
Robots trained via Reinforcement Learning (RL) or Imitation Learning (IL) often adapt slowly to new tasks, whereas recent Large Language Models (LLMs) and Vision-Language Models (VLMs) promise knowledge-rich planning from minimal data. Deploying LLMs/VLMs for motion planning, however, faces two key obstacles: (i) symbolic plans are rarely grounded in scene geometry and object physics, and (ii) model outputs can vary for identical prompts, undermining execution reliability. We propose ViReSkill, a framework that pairs vision-grounded replanning with a skill memory for accumulation and reuse. When a failure occurs, the replanner generates a new action sequence conditioned on the current scene, tailored to the observed state. On success, the executed plan is stored as a reusable skill and replayed in future encounters without additional calls to LLMs/VLMs. This feedback loop enables autonomous continual learning: each attempt immediately expands the skill set and stabilizes subsequent executions. We evaluate ViReSkill on simulators such as LIBERO and RLBench as well as on a physical robot. Across all settings, it consistently outperforms conventional baselines in task success rate, demonstrating robust sim-to-real generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。