让机器人通过自我反思快速学会新任务,无需人工干预。
Reflection-Based Task Adaptation for Self-Improving VLA
- 用失败分析自动生成密集奖励,加速策略学习
- 成功轨迹引导模仿,防止奖励欺骗问题
- 适合需要自主适应的新场景机器人应用
预训练的视觉-语言-动作(VLA)模型推动了通用机器人的发展,但如何在不依赖人类干预的情况下高效适配新任务仍是难题。本文提出反射式自适应框架,构建自提升闭环:通过失败驱动的强化学习路径,利用视觉语言模型的因果推理能力,自动从失败中合成针对性的密集奖励函数,显著加速策略探索;同时引入成功驱动的质量引导监督微调路径,筛选高质量成功轨迹进行模仿,并结合条件课程机制辅助初始探索,确保策略始终对齐真实任务目标。实验在复杂操作任务中验证,结果表明该框架收敛更快、最终成功率更高,为实现高效可靠的自适应智能体提供了有效方案。
原文摘要 · Abstract (English)
Pre-trained Vision-Language-Action (VLA) models represent a major leap towards general-purpose robots, yet efficiently adapting them to novel, specific tasks in-situ remains a significant hurdle. While reinforcement learning (RL) is a promising avenue for such adaptation, the process often suffers from low efficiency, hindering rapid task mastery. We introduce Reflective Self-Adaptation, a framework for rapid, autonomous task adaptation without human intervention. Our framework establishes a self-improving loop where the agent learns from its own experience to enhance both strategy and execution. The core of our framework is a dual-pathway architecture that addresses the full adaptation lifecycle. First, a Failure-Driven Reflective RL pathway enables rapid learning by using the VLM's causal reasoning to automatically synthesize a targeted, dense reward function from failure analysis. This provides a focused learning signal that significantly accelerates policy exploration. However, optimizing such proxy rewards introduces a potential risk of "reward hacking," where the agent masters the reward function but fails the actual task. To counteract this, our second pathway, Success-Driven Quality-Guided SFT, grounds the policy in holistic success. It identifies and selectively imitates high-quality successful trajectories, ensuring the agent remains aligned with the ultimate task goal. This pathway is strengthened by a conditional curriculum mechanism to aid initial exploration. We conduct experiments in challenging manipulation tasks. The results demonstrate that our framework achieves faster convergence and higher final success rates compared to representative baselines. Our work presents a robust solution for creating self-improving agents that can efficiently and reliably adapt to new environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。