AHA让机器人用自然语言识别并分析操作失败,提升真实场景适应能力。
AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
- 将失败检测转化为自由文本推理任务,通过语言模型生成可适配的解释。
- 在多个数据集上超越现有模型35.3%,真实场景任务成功率平均提升21.4%。
- 开源模型+自动生成失败数据集,适合需自主纠错的机器人系统研究者。
开放世界中的机器人操作不仅需要完成任务,还需具备失败识别与学习能力。尽管视觉-语言模型(VLM)和大语言模型(LLM)提升了机器人的空间推理与问题解决能力,但对失败的识别仍存在困难。本文提出AHA,一个开源的视觉-语言模型,通过自然语言实现对机器人操作失败的检测与推理。将失败检测建模为自由形式的推理任务,AHA能识别失败并提供跨不同机器人、任务和环境的详细、可调整的解释。我们使用FailGen——一个可扩展的框架,通过程序化扰动仿真中的成功示范,生成首个大规模机器人失败轨迹数据集(AHA数据集),用于微调AHA。尽管仅在该数据集上训练,AHA在真实世界失败数据集、多种机器人系统及未见过的任务上均表现良好,性能比次优模型GPT-4o高10.3%,超过六种对比模型(含五种顶尖VLM)平均35.3%。我们将AHA集成至三个基于LLM/VLM的操控框架中,分别用于强化学习、任务与运动规划、零样本轨迹生成。其失败反馈优化了密集奖励函数、任务规划流程与子任务验证,使三类任务的平均成功率较GPT-4模型提升21.4%。
原文摘要 · Abstract (English)
Robotic manipulation in open-world settings requires not only task execution but also the ability to detect and learn from failures. While recent advances in vision-language models (VLMs) and large language models (LLMs) have improved robots' spatial reasoning and problem-solving abilities, they still struggle with failure recognition, limiting their real-world applicability. We introduce AHA, an open-source VLM designed to detect and reason about failures in robotic manipulation using natural language. By framing failure detection as a free-form reasoning task, AHA identifies failures and provides detailed, adaptable explanations across different robots, tasks, and environments. We fine-tuned AHA using FailGen, a scalable framework that generates the first large-scale dataset of robotic failure trajectories, the AHA dataset. FailGen achieves this by procedurally perturbing successful demonstrations from simulation. Despite being trained solely on the AHA dataset, AHA generalizes effectively to real-world failure datasets, robotic systems, and unseen tasks. It surpasses the second-best model (GPT-4o in-context learning) by 10.3% and exceeds the average performance of six compared models including five state-of-the-art VLMs by 35.3% across multiple metrics and datasets. We integrate AHA into three manipulation frameworks that utilize LLMs/VLMs for reinforcement learning, task and motion planning, and zero-shot trajectory generation. AHA's failure feedback enhances these policies' performances by refining dense reward functions, optimizing task planning, and improving sub-task verification, boosting task success rates by an average of 21.4% across all three tasks compared to GPT-4 models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。